A visual object tracking method includes the steps of: setting an object window having a target in a video image; defining a search window greater than the object window; analyzing an image pixel of the object window to generate a color histogram for defining a color filter which includes a dominant color characteristic of the target; using the color filter to generate an object template and a dominant color map in the object window and the search window respectively, the object template including a shape characteristic of the target, the dominant color map including at least one candidate block; comparing the similarity between the object template and the candidate block to obtain a probability distribution map, and using the probability distribution map to compute the mass center of the target. The method generates the probability map by the color and shape characteristics to compute the mass center.