An image edge enhancement apparatus and a method thereof used for adjusting the brightness of a pixel are provided. The image edge enhancement apparatus comprises a first edge judgment circuit, a second edge judgment circuit and a process circuit. The first edge judgment circuit retrieves the edge data of the pixel in a first direction to output a first edge value. The second edge judgment circuit retrieves the edge data of the pixel in a second direction to output a second edge value. The process circuit determines an operation mode according to the brightness of a plurality of adjacent pixels around the pixel, the first edge value and the second edge value to output an adjusted brightness of the pixel according to operation mode. The adjusting method effectively avoids the noise in an image being amplified, and avoids the same edge being repetitively enhanced by different edge judgment circuits.