A moving feature is recognized in a video sequence by comparing its movement with a characteristic pattern. Possible trajectories through the video sequence are generated for an object by identifying potential matches of points in pairs of frames of the video sequence. When looking for the characteristic pattern, a number of possible trajectories are analyzed. The possible trajectories may be selected so that they are suitable for analysis. This may include selecting longer trajectories that can be easier to analyze. Thereby where the object being tracked is momentarily behind another object a continuous trajectory is generated.