An apparatus for and a method of processing a document image are provided. The method comprises: generating a luminance component image from the document image; estimating a luminance image from the luminance component image; and adjusting the luminance component image according to the estimated luminance image. Luminance values of pixels at least in horizontal edge areas of the luminance component image are estimated according to luminance values of pixels in a part of background of the luminance component image. If the estimated luminance values are acceptable according to a predetermined criterion, the luminance image is estimated according to the estimated luminance values. If the estimated luminance values are unacceptable, the luminance image is estimated by using the largest one of the luminance values of the pixels in each column of pixels in the luminance component image as the luminance values of all of the pixels in the column.