A method for tracking an object across a number of image frames comprises identifying a region containing the object in a first image frame to be stored as an exemplar view of the object. An appearance model (modified Exemplar View histogram is created based on the region in the first image frame and a background region in a second image frame, and the method determines at least one of a location and size of a predicted region for tracking the object in the second image frame using the appearance model. The method corrects at least one of the determined location and size of the predicted region in the second image frame in accordance with at least one of the location and size of the region in the first image frame corresponding to the exemplar view of the object.