A mechanism is provided in a data processing system for shared buffer affinity for multiple ports. The mechanism configures a physical first-in-first-out (FIFO) buffer with a plurality of FIFO segments associated with a plurality of network ports. The plurality of network ports share the physical FIFO buffer. The mechanism identifies a FIFO segment under stress within the plurality of FIFO segments. The mechanism reconfigures the physical FIFO buffer to assign a portion of buffer space from a FIFO segment not under stress within the plurality of FIFO segments to the FIFO segment under stress.