Visual tracking over a sequence of images is formulated by defining an object class and one or more background classes. The most discriminant features available in the images are then used to select a portion of each image as belonging to the object class. Fisher's linear discriminant method is used to project high-dimensional image data onto a lower-dimensional space, e.g., a line, and perform classification in the lower-dimensional space. The projection function is incrementally updated.