A method and apparatus for controlling a motion vector buffer for encoding or decoding of a video sequence includes identifying a set of motion vectors associated with an above coding tree unit (CTU), wherein each motion vector is associated with a P×Q grid, and the set of motion vectors is associated with an N×M grid. A motion vector is determined based on the set of motion vectors. The motion vector, in the motion vector buffer, is accessed based on a candidate block including a position associated with the N×M grid. A current CU can search from each 4×4 position for a current block. When the search range is beyond the current CTU, the motion vector (MV) data from the last row of the above CTU is used.