An image processing apparatus generates second image data having different numbers of pixels from image data having first resolution for each frame, outputs first and second image data in a first order, subjects the first and second image data to a predetermined process in a second order which is different from the first order, and controls the writing of the output image data in a memory and the readout of the processed image data from the memory by the unit of the frame, according to the first and second order. The read address in the first frame and the write address of the second frame which is subsequent to the first frame are an address of a first storage region of the memory, and the write address in the second frame is determined to be the read address according to which the first frame has been read out.