An image processing apparatus and method thereof are provided. The image processing apparatus stores at least a reference image and performs the following operations: (a) receiving an image, (b) determining a plurality of representative keypoints for the image, such as determining the representative keypoints by a density restriction based method, (c) finding out that a matched area in the image corresponds to a first reference image according to the representative keypoints, (d) determining that a matched number between the representative keypoints and a plurality of reference keypoints of the first reference image is less than a threshold, and (e) storing the matched area in the image processing apparatus as a second reference image.