A method of motion estimation (ME) refinement. The method generally includes the steps of (A) generating an initial motion vector (MV) by conducting a first ME on an initial block in a picture, the initial block covering an initial area of the picture, (B) generating a current MV by conducting a second ME on a current block in the picture, (i) the current block covering a subset of the initial area and (ii) the second ME being seeded by the initial MV, (C) generating at least one additional MV by conducting at least one third ME on the current block, the at least one third ME being seeded respectively by at least one neighboring MV spatially adjacent to the current MV and (D) generating a refined MV of the current block by storing in a memory a best among the current MV and the additional MV.