A cutting tool holding device includes a base, a cutting tool holder and a fastening member. The base includes an abutting portion and a transverse dovetail groove, wherein the abutting portion is defined between a first wall with a mounting surface and a second wall, and the transverse dovetail groove is formed on the mounting surface. The cutting tool holder has a transverse dovetail portion, wherein the cutting tool holder is abutted against the abutting portion, and the transverse dovetail portion is detachably engaged into the transverse dovetail groove on the mounting surface of the second wall. The fastening member is for fastening the cutting tool holder to the base.