An image processing apparatus includes a memory configured to store information of an OTF or a PSF of an optical system for at least one of capturing conditions, and an image processor configured to generate secondary and higher components of a phase of an OTF or a shape component of a PSF corresponding to a capturing condition of a captured image through an interpolation based on at least two OTFs or PSF in the memory, while center-of-gravity positions or maximum intensity positions are accorded with each other or differential root-mean-square values of the point spread functions are minimum, and to restore the image utilizing the generated OTF or an OTF derived from the generated PSF.