Systems and methods for tracking an object in video surveillance are provided. A method may include obtaining a video including a plurality of consecutive frames; obtaining a current frame from the plurality of consecutive frames, wherein an object of interest is identified in at least two previous frames of the current frame; obtaining at least two template frames from the at least two previous frames; and identifying a position related to the object of interest in the current frame based on the at least two template frames using a correlation filter.