The present image processing apparatus mutually converts raster-order image data and block-order image data by dividing image data into a plurality of areas in units of bands, and performing reading-out and writing of the image data to one line memory in the units of bands of the division. The apparatus calculates, in the units of bands, an address increased amount, which is used for calculating an address to next write to or read out from, from an address that is a current processing target, calculates an address of a write destination when writing image data to the line memory and an address of a readout destination when reading-out image data from the line memory by adding the calculated address increased amount to an address value of a current processing target address.