Embodiments described herein provide examples of a real-time visual object tracking system. In one aspect, an unmanned aerial vehicle (UAV) capable of performing real-time visual tracking of a moving object includes: a processor; a memory coupled to the processor; and a camera to capture a video of the moving object. This UAV additionally includes a visual tracking module to: receive a first video image and a first location of the object; receive a second video image following the first video image; place a first search window in the first video image and a second search window in the second video image centered on a second location in the second video image having the same coordinates as the first location; compute a correlation between an image patch within the first search window and an image patch within the second search window; and determine an updated location of the object in the second video image.