A quick-release bit adapter includes a body having first and second ends spaced along a longitudinal axis. The body includes a driving hole extending from the first end towards the second end along the longitudinal axis. The body further includes a sliding groove in communication with the driving hole and intermediate the driving hole and the second end along the longitudinal axis. A through-hole extends from an outer periphery of the first end through the sliding groove. A bit received in the driving hole abuts and is stopped by at least one protrusion extending inward from an inner periphery of the driving hole towards the longitudinal axis. A magnetic device is slideably received in the sliding groove and controllable by a user through the through-hole to slide along the longitudinal axis between two positions to engage with or disengage from the bit.