An image processing apparatus divides a plurality of difference images that are based on addition and subtraction between a plurality of parallax images, and a second difference image between a current composite image that is based on addition of the plurality of parallax images and a past composite image, into a plurality of sub-bands. As for the plurality of first difference images, the image processing apparatus then encodes data of a sub-band that includes high-frequency components, out of the plurality of sub-bands. As for the second difference image, the image processing apparatus encodes data of the plurality of sub-bands.