A process performs a search on a base reference frame in a video signal to determine a first motion vector for each of the possible partitions of a current macroblock. The first motion vector for a partition provides the most optimal match for the partition in the base reference frame. An optimal partition for the current macroblock, which gives a minimum motion estimation cost among all the possible partitions in the base reference frame, is determined. The process performs the search on the optimal partition over a non-base reference frame in the video signal to determine a second motion vector that provides the most optimal match for the corresponding partition in the non-base reference frame. A simplified search is performed on each partition other than the optimal partition over the non-base reference frame to determine the second motion vector for the corresponding partition.