Systems and methods are described for robust online face tracking. In one implementation, a system derives multiple resolutions of each video frame of a video sequence portraying movement of a visual object. The system tracks movement of the visual object in a low resolution as input for tracking the visual object in a higher resolution. The system can greatly reduce jitter while maintaining an ability to reliably track fast-moving visual objects.