A video coder is configured to determine bi-directional motion vectors of a current block of the video data and determine that a condition is satisfied with respect to the current block based on each component of the bi-directional motion vectors of the current block being less than a threshold value. The video coder is further configured to, based on the condition being satisfied with respect to the current block, early terminate application of a motion vector refinement process to the bi-directional motion vectors of the current block. The video coder is further configured to determine a prediction block for the current block based on the bi-directional motion vectors of the current block and reconstruct the current block based on the prediction block for the current block.