A focus detection device calculates a shift amount between a first focus detection signal and a second focus detection signal both generated based on a signal obtained from an image sensor. The device also calculates a conversion coefficient and applies the conversion coefficient to the shift amount to convert the shift amount into a defocus amount of an imaging optical system. The device calculates the conversion coefficient based on at least one of a pupil eccentricity amount and an incident pupil distance, both are dependent on an image height at a focus detection position.