A method (199) is disclosed for modifying colors of text extracted from an image of a document page to improve visual quality. The document image comprises text and image regions of various colors. The method divides (100) the document image into a plurality of connected components, the connected components comprising pixels of similar color that are closely spatially located, and selects from the image a plurality of the connected components that represent text. The plurality of text connected components are grouped to form a logical structure grouping of said page comprising one or more text blocks wherein each said text block represents a text line or paragraph. One or more logically consistent color sets are then formed (120, 699, 399, 499) for each text block based on the colors of the connected components in the text block. The logically consistent color sets represent all text of visually similar color of the corresponding text line or paragraph. The method then applies color enhancement (130) to the logically consistent color sets to enhance visual quality and correct image defects in the document image.