A control apparatus includes a first calculating unit configured to calculate a defocus amount by a focus detection of a phase difference method, a second calculating unit configured to calculate a contrast evaluation value of an image, a third calculating unit configured to calculate a function indicating a relationship between the defocus amount and the contrast evaluation value calculated at each of a plurality of lens positions, and a fourth calculating unit configured to calculate an adjusting value of the defocus amount based on an extreme value of a function.