A motion estimation method for video compression comprises the following steps. First, an initial simplex comprising three points is determined based on motion vectors in blocks of a current frame and a previous frame, and a point having a largest function value among the three points is replaced with a point having a smaller function value to form a simplex. The replacement is repeated until two points of the three points of the simplex converge to a same point. The iteration is performed by downhill simplex search including operations of reflection, expansion, contraction and shrinkage to find a point for replacement. The motion estimation method for video compression can also use multi-reference frames. An initial simplex comprising four points is determined based on motion vectors of a current frame with reference to a plurality of previous frames, and a point having a largest function value among the four points is repeatedly replaced with a point having a smaller function value to form a simplex until two points of the four points of the simplex converge to a same point.