A method of processing a video sequence is provided that includes tracking a first object and a second object for a specified number of frames, determining similarity between a trajectory of the first object and a trajectory of the second object over the specified number of frames, and merging the first object and the second object into a single object when the trajectory of the first object and the trajectory of the second object are sufficiently similar, whereby an accurate location and size for the single object is obtained.