This invention is directed to efficiently compressing data used in an image process for an unprocessed line adjacent to a process line in an image data process for each line, and reducing the memory band. To accomplish this, processed image data is compressed, and it is determined whether the process direction of image data of the next line is the first direction or the second direction opposite to the first direction. The data organization of the compressed image data is changed in accordance with the determination result. A memory stores the compressed image data whose data organization has been changed. The compressed image data stored in the memory is read out and decompressed. An image process is performed using the decompressed image data and image data of a line adjacent to the line of the processed image data.