An image processing apparatus comprises: an unit configured to sequentially determine, as a processing object, each of blocks, and to compare color data of respective pixels in the block that is the processing object, thereby specifying a pattern flag indicating an arrangement pattern of color data; an unit configured to extract first color data from a pixel, and further to extract second color data to fourth color data corresponding to the arrangement pattern; an unit configured to hold the pattern flag of each of the blocks, the first color data, and the second color data to fourth color data; and an unit configured to specify addresses in a memory, at which the first color data and the second color data to fourth color data in a block to be composed are held.