Embodiments are directed towards tracking physical objects' pixel locations in a stream of video frames. Radio frequency (RF) readers may be positioned relative to a scene. RF tags may be positioned in the scene at known pixel locations within the field of view of a video recording device. The RF tags may be enabled to generate RF signals to the RF readers. The RF readers may generate RF values based on the RF signals, which may be employed to determine a function to translate the RF values into known pixel locations within a video frame. An RF tag may be disposed at a physical location of at least one object within the scene. A stream of video frames and RF values may be recorded over time. The function may be employed to translate the recorded RF values into pixel locations for the least one object each recorded video frame.