An image processing apparatus processes image signals obtained from an image pickup element. A correlation operation unit operates a correlation of the image signals of two images that are obtained from the image pickup element and operates an image shift amount between images that are picked up by photoelectric conversion units, on the basis of the correlation. A saturation detection unit detects whether an image signal, of the two images, reaches a saturation level, and produces a saturation detection result. A controller controls the correlation operation unit such that the controller controls whether the correlation of the image signals from a plurality of photoelectric conversion units, including the photoelectric conversion units, is used for the operation of the image shift amount performed by the correlation operation unit. The control by the controller is made in accordance with the saturation detection result produced by the saturation detection unit.