Invention describes a method and system for detecting and tracking an object in a sequence of images. For each image the invention determines an object descriptor from a tracking region in a current image in a sequence of images, in which the tracking region corresponds to a location of an object in a previous image. A regression function is applied to the descriptor to determine a motion of the object from the previous image to the current image, in which the motion has a matrix Lie group structure. The location of the tracking region is updated using the motion of the object.