Illumination information indicating the center of illumination and lightness change direction is acquired from an image, and the image is divided into regions for respective colors based on the acquired illumination information and color information of respective pixels included in the image. Especially, thresholds in clustering are set step by step in accordance with the lightness change direction from the center of the illumination information, and the image is clustered to the regions for respective colors according to the thresholds which are set step by step.