A tool transfer system configured to transfer a tool between first and second tool holding devices by causing the first and second tool holding devices to be moved relative to each other. The first tool holding device includes a clamp sleeve forced in a direction toward the second tool holding device so as to be positioned in a clamping position. The second tool holding device includes: a hole defining member defining a receiving hole that is to receive the tool; and a movable engaging member that is movable between a working position and a non-working position depending on movement of the hole defining member. The movable engaging member is engageable with the clamp sleeve, while being positioned in the working position, such that the clamp sleeve can be moved, by engagement of the movable engaging member with the clamp sleeve, so as to be positioned in an unclamping position, during relative axial movement of the first and second tool holding devices toward each other.