A tool adapter includes a body and a retaining device. The body has a coupling hole and a side hole interconnected with the coupling hole. The retaining device includes an outer sleeve and a retaining member. The outer sleeve is movable relative to the body and is provided with an interacting portion. The retaining member has an engaging portion arranged in the side hole, a first portion arranged on an outer periphery of the body, and a bend portion located between the engaging portion and the first portion. The interacting portion releasably abuts against the bend portion in response the movement between the outer sleeve and the body.