A method for high performance caching is disclosed. The method generally includes steps (A) and (B). Step (A) may fetch a plurality of reference samples of a reference image from a first circuit to a cache of a second circuit. The cache may include a plurality of cache blocks and a plurality of valid bits. Each of the cache blocks generally corresponds to at most one of the valid bits. A size of the cache blocks may match a smallest read access size of the first circuit. Step (B) may transfer the reference samples having the corresponding valid bit set to valid from the cache to a processor of the second circuit.