A tool retaining device includes a housing having an engaging hole and having a chamber for slidably receiving a tool stem, the tool stem includes a socket opening for receiving a tool member which is also slidably engaged in the engaging hole of the housing, for allowing the tool member to be rotated by either the tool stem or the housing. The tool member may also be moved relative to the housing for changeably coupling tool members or tool bits having different lengths, and may be selectively moved into the housing, for allowing another tool element to be selectively engaged with the engaging hole of the housing when the tool member is moved into the housing.