An imaging unit is configured with use of an imaging element including an image output pixel and a phase difference detection pixel. A control unit performs focus control based on image-plane phase difference information obtained using the phase difference detection pixel over a predetermined period, generates focus state change information indicating whether or not a change in a focus state is a monotonous change, and continues the focus control based on the image-plane phase difference information on the assumption of a large blur state in a case where the focus state change information indicates that a change in the focus state is a monotonous change. The control unit switches from the image-plane phase difference auto-focus operation to an auto-focus operation in a case where a change in the focus state is not a monotonous change. Thus, it becomes possible to perform a high-speed and high-quality auto-focus operation.