Disclosed are various embodiments for tracking an object shown as moving in a video. One embodiment is a method for tracking an object in a video that comprises tracking in a first temporal direction an object in a plurality of video frames and generating a first tracking result, evaluating the first tracking result corresponding to tracking of the object in the first temporal direction, and stopping tracking in the first temporal direction upon the occurrence of a predefined event, wherein the predefined event is based on an evaluated tracking result. The method further comprises obtaining data identifying an object outline of the object upon stopping the tracking in the first temporal direction, tracking in a second temporal direction the object based on the data identifying the object outline of the object to generate a second tracking result, and generating a refined tracking result based on at least on one of the first tracking result, the second tracking result, or a combination thereof.