A method for decoding a current block in a current picture of a video bitstream includes decoding, from the video bitstream, a first motion vector for the current block relative to a first reference block of a first reference picture having a first picture order count, and decoding, from the video bitstream, a second motion vector for the current block relative to a second reference block of a second reference picture having a second picture order count. A similarity metric is generated based on a comparison of the first motion vector and the second motion vector. The method further includes determining whether to refine the first motion vector based on the similarity metric, generating a first refined motion vector from the first motion vector, and performing motion compensation to derive a first reference block from the first reference picture using the first refined motion vector.