A method, system and computer program product for the selection of reference pictures and mode decision during the motion estimation of a video sequence is disclosed. The video sequence includes a plurality of video frames. Each of the video frames includes a plurality of macroblocks. A full-pixel search is performed for each of the video frames, to calculate one or more full-pel rate distortion costs. Based on the one or more rate-distortion costs, one of the one or more modes is selected. Further, interpolation is performed for each of the video frames. Thereafter, a sub-pixel search is performed for each of the video frames, to calculate the one or more sub-pel rate distortion costs. Finally, a reference video frame is selected, based on the one or more sub-pel rate distortion costs.