A tool holder has a chuck portion with a retaining bore in which a cutting tool can be inserted. The tool has flat surfaces extending from its end, and a circumferential groove at rear ends of the flat surfaces. When the tool is inserted into the retaining bore, first protrusions of the tool holder pass through spaces between the flat surfaces and the inner wall of the retaining bore and fitted in the groove. When the tool is turned, the protrusions move in and engage the side edges of the groove, thus preventing the tool from pulling out of the chuck portion. When the first protrusions are fitted in the groove, second protrusions of the tool holder are located on the flat surfaces. When the tool is turned, the second protrusions are received in ends of the respective spaces, thereby preventing rotation of the tool relative to the chuck portion.