The focus detection apparatus includes an image pickup part 107 including first and second pupil division pixels SHA and SHB to produce first and second image signals, a processor 121 performing a restoration process on the first and second image signals to produce first and second restored image signals, and a calculating part calculating a provisional value of a defocus amount of an image-forming optical system by using the first and second image signals. When the provisional value is smaller than a predetermined value, the restoration process is performed to produce pluralities of the first and second restored image signals by using a greater number of image restoration filters than when the provisional value is greater than the predetermined value, and the defocus amount is calculated by using the pluralities of the first and second image signals.