A scaled motion search section can be used in a video processing device that processes a video input signal that includes a plurality of pictures. The scaled motion search section includes a downscaling module that downscales the plurality of pictures to generate a plurality of downscaled pictures, wherein the downscaling module includes a horizontal downscaling filter and a vertical downscaling filter, and wherein the vertical downscaling filter generates downscaled pixels for a macroblock pair using only pixels from the macroblock pair. A transfer function that models the scaled motion vectors is determined and used to identify a final set of motion vector candidates used in a larger scale motion search.