The present invention provides a framework for the processing of blocks between two data frames and in particular application to motion estimation calculations in which a balance among the performance of a motion search algorithm, the size of on-chip memory to store the reference data, and the required data transfer bandwidth between on-chip and external memory can be optimized in a scalable manner, such that the total system cost with hierarchical embedded memory structure can be optimized in a flexible manner. The scope of the present invention is not limited to digital video encoding in which motion vector is part of information to be encoded, but is applicable to any other implementation in which difference between any two data frames are to be computed.