An image processing apparatus that enables to reduce needless consumption of memory band and control duplicated access to a main memory. A reading unit reads image data stored in a first storage unit and divides the image data into a plurality of rectangular areas of a predetermined size. A second storage unit stores image data in reference areas surrounding the rectangular areas, the reference areas having overlapped areas each of which includes a boundary between adjacent two rectangular areas. An image processing unit performs an image process based on the image data in the rectangular areas read by the reading unit and the image data in the reference areas stored in the second storage unit. A cache control unit controls to transfer the image data in the reference areas from the second storage unit to the image processing unit in response to a request from the image processing unit.