A method tracks an object in a sequence of frames of a video. The method is provided with a set of tracking modules. Frames of a video are buffered in a memory buffer. First, an object is tracked in the buffered frames forward in time using a selected one of the plurality of tracking module. Second, the object is tracked in the buffered frames backward in time using the selected tracking module. Then, a tracking error is determined from the first tracking and the second tracking. If the tracking error is less than a predetermined threshold, then additional frames are buffered in the memory buffer and the first tracking, the second tracking and the determining steps are repeated. Otherwise, if the error is greater than the predetermined threshold, then a different tracking module is selected and the first tracking, the second tracking and the determining steps are repeated.