A focus detection apparatus comprises a line sensor including a photoelectric conversion element and an integrating capacitor, a storage unit configured to store information concerning a defective pixel, a first comparison unit configured to compare a signal from the defective pixel with a predetermined threshold, a switching unit configured to switch between a first accumulation mode and a second accumulation mode, and a control unit configured to control the switching unit so as to set the line sensor in the first accumulation mode if the first comparison unit determines that a signal from the defective pixel is not more than the predetermined threshold and set the line sensor in the second accumulation mode if the first comparison unit determines that a signal from the defective pixel is larger than the threshold.