An image processing apparatus divides an image into a plurality of first image blocks, each of which is a partial area of the image, and further divides each first image block to acquire a plurality of compartments that correspond to filters for determining an edge direction of an image of the first image block. The image processing apparatus determines the edge direction of each first image block based on the plurality of compartments and the filters, and decides that a first image block whose edge direction cannot be determined is a failed image block. The image processing apparatus then generates, from the failed image block, a second image block constituted by a plurality of compartments that differ in size from the plurality of acquired compartments, and determines the edge direction of the second storage block based on the plurality of newly generated compartments and the filters.