A bit retaining system for a rotary tool or device such as a router is disclosed. The bit retaining system may include a collet assembly and a control assembly. The collet assembly may include a spindle, a collet adapted to receive the bit, and a collet nut. The collet assembly may further include a collet gear mounted on the collet nut and/or a spindle gear mounted on the spindle. The control assembly selectively engages the collet assembly to drive the collet nut and/or secure the spindle in a fixed position, preventing its rotation. In operation, a hand tool is utilized to engage the control assembly. This configuration enables a user to capture and release the rotary tool bit using one hand.