Aspects of the present invention relate to systems and methods for refining text color in a digital image. A pixel of interest from a first plurality of pixels in a digital image may be selected, and the first plurality of pixels may be separated into a second plurality of pixels and a third plurality of pixels based on the color value of the pixel of interest. A refinement color value may be calculated based on the second plurality of pixels when the second plurality of pixels meets a quantity threshold. When the second plurality of pixels does not meet the quantity threshold, the refinement color value may be calculated based on the third plurality of pixels. The refinement color value may be assigned to the pixel of interest.