A method and apparatus of data reduction of search range buffer for motion estimation or motion compensation are disclosed. The method and apparatus use local memory to store reference data associated with search region to reduce system bandwidth requirement and use data reduction to reduce required local memory. The data reduction technique is also applied to intermediate data in a video coding system to reduce storage requirement associated with intermediate data. The data reduction technique is further applied to reference frames to reduce storage requirement for coding system incorporating picture enhancement processing to the reconstructed video.