A driver's physical condition detection device for detecting a physical condition of a driver driving a vehicle, includes: a vehicle detector configured to detect a change in motion of the vehicle during driving; a driver detector configured to detect a change in motion of the driver; a calculator configured to calculate a follow-up degree of the change in motion of the driver with respect to the change in motion of the vehicle during driving; and a physical condition determination portion configured to perform a determination process of determining whether or not the physical condition of the driver is deteriorated, based on the follow-up degree.