The defocus amount of an imaging optical system is detected based on a pair of image signals that are obtained from an image sensor and are used for phase-difference detection automatic focusing. In a first mode, the drive of a focusing lens is controlled so that the focusing lens is driven at a first speed if the reliability of the pair of image signal is at a first level, and at a second speed lower than the first speed if the reliability of the pair of image signals is at a second level higher than the first level.