Systems and methods for managing the volume of multiple VoIP streams are disclosed. The system includes a VoIP server configured to receive an input audio stream from a first VoIP handset, create separate output audio streams from the input audio stream for transmission to second and third VoIP handsets, and to connect to a communications network. The system also includes a volume control table coupled to the VoIP server, the volume control table including records of volume adjustments made during prior conversations between the two or more VoIP handsets. The VoIP server is further configured create the separate output audio streams such that one or more of the output streams has a volume that is different than input audio stream based on the records.