Method and apparatus for selecting a reference frame for motion estimation in a video encoder is described. In one example, motion estimation is performed on a current macroblock in a current frame with respect to a reference frame to obtain a motion vector, a sum of absolute difference (SAD), a global motion vector, and an average SAD. A first relative change of the motion vector with respect to the global motion vector is computed. A second relative change of the SAD with respect to the average SAD is computed. A functional based on the first relative change, the second relative change, a first parameter, and a second parameter is computed. The functional is compared with a threshold. The reference frame is selected if the functional satisfies the threshold.