A bit holder assembly for a rotary hand or powered tool, includes a body having a hex shank at its rearward end and a retraction collar slidably disposed on the body. The body has a coaxial hex socket formed therein to allow a tool bit to be inserted thereinto. A coil spring biases the retraction collar forwardly and biases a retaining clip toward the bottom of an angular slot formed in the body. The bit is removed by sliding the retraction collar rearwardly, to compress the coil spring and allow the retaining clip to retract back up the slot. A ring magnet is disposed at the forward end of the retraction collar to magnetize a fastener to retain the fastener to the bit.