A method and apparatus for encoding of a video sequence in an encoder or decoding of the video sequence in a decoder includes parsing an initial motion vector from the video sequence associated with a block. A plurality of samples are determined and pre-fetched to permit both motion vector refinement and motion compensation based on parsing the initial motion vector. Motion vector refinement is performed to determine a final motion vector using a first subset of the plurality of samples, and motion compensation is performed using a second subset of the plurality of samples.