Methods and apparatus that may be utilized to reduce latency associated with encryption based on externally stored security metadata are provided. When encrypted data is accessed for the first time, a cache line containing corresponding metadata used for decryption may be placed in an internal security metadata cache. If that data is accessed again, it may be retrieved without accessing external memory, thus reducing latency. Further, if adjacent data is accessed, the cached line may contain sufficient metadata to decrypt the adjacent data. As a result, a separate operation to access metadata for the adjacent data may be avoided, thus reducing latency.