A motion vector refining apparatus including a control unit, N filter units, and a mixer unit is provided. The control unit receives a motion estimation signal generated by a motion estimation unit and calculates a plurality of variation parameters according to a current motion vector, surrounding motion vectors, or a block matching error, so as to generate a control signal. The N filter units respectively calculate N filtering motion vectors by using N analysis processes. The mixer unit weights and mixes the filtering motion vectors according to the control signal to adjust and output a refined motion vector. Thereby, the motion vector refining apparatus can detect the edge of a moving object and mix filtering results of the analysis processes to adjust and refine motion vectors, so as to reduce image defects caused by a smooth processing.