The invention provides a video processing method, which includes: performing motion estimation upon the target image block and a plurality of reference image blocks to calculate a plurality of block matching differences; determining a minimum block matching difference from the plurality of the block matching differences; comparing the minimum block matching difference with at least one of the reference block matching differences to obtain at least a comparison result; and according to the at least a comparison result, selectively determining a target motion vector of the target image block referred to a motion vector of at least a neighboring image block of the target image block or a motion vector corresponding to the minimum block matching difference.