A signal processing apparatus that can suppress degradation of accuracy of phase difference detection. An obtaining unit obtains a plurality of frames of image signals from a plurality of photoelectric conversion units, which receives light fluxes with different incident directions from an object, an information receiving unit receives saturation information indicating whether the obtained image signals are saturated, a filter arithmetic unit subjects the output image signals of the plurality of frames to filter processing, an evaluation value calculation unit calculates a multivalued saturation evaluation value indicating reliability of the image signals subjected to the filter processing using the saturation information of the image signals of the frames most recently output, and a phase difference detection unit determines whether to use the image signals subjected to the filter processing for phase difference detection based on the calculated saturation evaluation value.