A focus detector, which detects a defocus amount from a displacement amount between images formed by a pair of light beams split from an image pickup system so as to pass through a pair of pupil regions, includes a pair of lenses and phase difference sensors, a memory unit for storing an image displacement amount between the image signals on the phase difference sensors in an in-focus state, a waveform read out controller for setting pixels to be calculated for the phase difference sensors, respectively, based on the image displacement amount, a correlation calculator for calculating a correlation amount between the image signals from the pixels to be calculated, a waveform degree-of-conformity calculator for calculating a waveform degree-of-conformity based on the image signals obtained from the pixels to be calculated, and a defocus calculator for calculating the defocus amount based on the correlation amount and the waveform degree-of-conformity.