An image processing apparatus including an input unit configured to input image data of a target frame, a first processing unit configured to perform a first processing for output image data of a reference frame related to the target frame, which is stored in the memory, to change positions of pixels in the output image data, an image processing unit configured to perform image processing for the image data of the target frame input by the input unit based on the output image data of the reference frame processed by the first processing unit, and a second processing unit configured to perform a second processing, the second processing being according to the first processing, for the image data of the target frame for which image processing has been performed by the image processing unit and to store the processed image data in the memory as output image data of the target frame.