An image processing apparatus converts a multivalued image to a binary image, determines a character region and a non-character region in the binary image, extracts areas in units of a character from the character region, determines whether each of the areas is a character region or a non-character region, and extracts, for each of the areas determined to be the character region, a first representative color from the multivalued image included in the area, and generates a binary image corresponding to the first representative color. The apparatus decides a second representative color closest to a color of a target pixel by comparing the first representative color with the color of the target pixel, and changes the binary image of the area corresponding to the first representative color to a binary image of the decided second representative color.