The invention provides an image color edge removing method and device. The method comprises the steps of carrying out scanning to obtain a test image; determining a reference color in the test image and taking a scanning line on which the reference color is located as a reference line; determining displacement quantity between non-reference colors and the reference color according to the referenceline; and carrying out displacement adjustment on the scanning lines of the non-reference colors according to the determined displacement quantity, thereby enabling the scanning lines on which the reference color and the non-reference colors are located after the displacement adjustment to be overlapped and obtaining a color edge removed scanned image. The color malposition of the color edges isaccurately identified and corrected, so an effect that scanned character or line edges are clear and an original scanning text is restored truly is achieved.