Methods and apparatuses of determining an alignment level between motion compensated reference patches for reducing motion vector refinement steps are provided. According to one method, obtaining, by a decoder, motion compensated interpolated samples based on sub-pixel accurate merge motion vectors from a bilinear motion compensated interpolation; computing, by the decoder, a sum of absolute differences (SAD) between two motion compensated reference patches using a subset of the motion compensated interpolated samples; determining, by the decoder, whether the SAD is less than a coding unit (CU) size-dependent threshold value; when the SAD is less than the CU size-dependent threshold value: skipping remaining decoder-side motion vector refinement (DMVR) process steps; and performing final motion compensation; and when the SAD is not less than the CU size-dependent threshold value: performing the remaining DMVR process steps; and performing the final motion compensation.