A cutting tool assembly includes a holder having a front face that defines a longitudinal bore extending at least partially through the holder, a cutting tool having a head portion, a shank portion and a collar portion therebetween all extending along a longitudinal axis of the cutting tool, a retainer sleeve substantially surrounding the shank portion and configured for being removably received in the longitudinal bore of the holder and an outwardly biasing retention member connected to the retainer sleeve and configured for cooperating with the longitudinal bore of the holder.