In a focus detection apparatus having a plurality of filters suitable for processing of signals having different frequencies, a setting unit sets a thinning rate in a main-scanning direction of an image sensor depending on a filter to be used, a control unit controls to read image signals sequentially with the set thinning rate and store the read image signals in a storage unit, a processing unit processes the stored image signals with the filter and obtains a focus state evaluation value in a sub-scanning direction of the image sensor and a focus control unit performs the focus control processing based on the focus state evaluation value. If the filter to be used is for a second frequency that is higher than a first frequency, the setting unit sets a lower thinning rate than that for a filter for the first frequency.