A control apparatus includes a focus detector configured to detect a defocus amount based on a phase difference between a pair of image signals generated based on light beams having passed through mutually different pupil areas in an imaging optical system, and a controller configured to control a focus lens in the imaging optical system. The controller is configured to perform an autofocus control based on the defocus amount detected by the focus detector, and a manual focus control based on an operation of a user via an operation member. The controller performs the autofocus control when a change amount relating to a speed of the focus lens in the manual focus control falls within a predetermined range.