A method of caching reference data in a reference data cache is provided that includes receiving an address of a reference data block in the reference data cache, wherein the address includes an x coordinate and a y coordinate of the reference data block in a reference block of pixels and a reference block identifier specifying which of a plurality of reference blocks of pixels includes the reference data block, computing an index of a set of cache lines in the reference data cache using bits from the x coordinate and bits from the y coordinate, using the index and a tag comprising the reference block identifier to determine whether the reference data block is in the set of cache lines, and retrieving the reference data block from reference data storage when the reference data block is not in the set of cache lines.