To achieve focusing suitable for a perspective-conflict object in AF control using a phase difference detection method, a focus adjustment apparatus detects a defocus amount of an image pickup optical system based on plural image signals generated from an exit pupil of the system, sets plural areas including a predetermined area used for focus adjustment of the system as areas of the plural image signals used for detecting the defocus amount, generates a focus adjustment signal for performing the focus adjustment based on the defocus amount detected in the predetermined area, and determines whether a perspective conflict occurs in the predetermined area according to the defocus amount detected in each set area. If the perspective conflict occurs in the predetermined area, the area used for the focus adjustment is changed from the predetermined area to any of other set areas, according to the defocus amount detected from each area.