A system and method are provided for object tracking in a sequence of images of a scene acquired by an imaging device. The method includes generating a representation of an object and its surroundings; generating a tracking score based on the representation of the object and its surroundings, and a classification scheme; detecting a peak in the tracking score corresponding to a location of the object in the image; and adjusting the representation of the object and its surroundings and the classification scheme according to one or more new appearances of the object.