A method and apparatus is provided for reconstructing video frames that include missing pixels as a result of video stabilization techniques to compensate for camera movement and/or zooming. In one example, the method and apparatus caches transformed frames of video, identifies coordinates of missing pixels in a current transformed frame, and sequentially processes, for only the missing pixel coordinates, the cached transformed frames in reverse chronological order to identify pixels at coordinates in the cached transformed frames having valid data and corresponding to one of the missing pixel coordinates. Upon identifying a pixel having valid data at a coordinate in a cached transformed frame corresponding to a missing pixel coordinate, the method and apparatus inserts the valid data at the missing pixel coordinate.