An image processing apparatus capable of generating a plurality of output images having different focus positions by reconstructing an input image, includes a storage unit storing image pickup condition information, and an image processing unit generating the output image from the input image using the image pickup condition information, and the image processing unit obtains the input image that is information of an object space viewed from a plurality of viewpoints that is obtained via an imaging optical system and an image pickup element having a plurality of pixels, calculates an average pixel value of a pixel group of the input image of the same region, and substitutes each pixel value of the pixel group by the average pixel value, and performs combination such that the pixels substituted by the average pixel value are shifted from each other to generate the output image.