An image processing apparatus has an acquisition unit adapted to acquire an image signal and an evaluation value signal corresponding to the image signal as a pair of signals; and a filter processing unit adapted to apply a filtering process to the pair of signals, wherein the filter processing unit applies a first filtering process to the image signal and applies a second filtering process to the evaluation value signal, in the first filtering process, refers to a first reference pixel to determine a pixel value of a first target pixel, and in the second filtering process, determines a pixel value of a second target pixel corresponding to the first target pixel based on a pixel value of a second reference pixel at a coordinate position corresponding to a coordinate position of the first reference pixel.