A method of processing video data includes: receiving a first video frame with first blocks of pixels associated with a known motion vector and a second video frame with second blocks of pixels, the second blocks corresponding to the first blocks, and uncovered blocks adjacent to the first frame boundary not corresponding to the first blocks in the first video frame; determining a first block for each line segment in the second video frame corresponding to a block in the first video frame, wherein the first block has the known motion vector from the first video frame; and assigning a motion vector for at least one of the line segments in the second video frame to the uncovered blocks in the line segment of the second video frame between first block and the first frame boundary.