A flow based reply cache of a storage system is illustratively organized into one or more microcaches, each having a plurality of reply cache entries. Each microcache is maintained by a protocol server executing on the storage system and is allocated on a per client basis. To that end, each client is identified by a client connection or logical “data flow” and is allocated its own microcache and associated entries, as needed. As a result, each microcache of the reply cache may be used to identify a logical stream of client requests associated with a data flow, as well as to isolate that client stream from other client streams and associated data flows used to deliver other requests served by the system. The use of microcaches thus provides a level of granularity that enables each client to have its own pool of reply cache entries that is not shared with other clients, thereby obviating starvation of entries allocated to the client in the reply cache.