A visual tracker may track an object by identifying the object in a frame, and the visual tracker by identify the object in the frame within a search region. The search region may be provided by a motion modeling system that independently models the motion of the object and models the motion of the camera. For example, an object motion model of the motion modeling system may first model the motion of the object, assuming the camera is not in motion, in order to identify the expected position of the object. A camera motion model of the motion modeling system may then update the expected position of the object, obtained from the object motion model, based on the motion of the camera.