A peer-aware voice stream ranking method that makes decisions based on information about participants of a voice conference over a network. Whether to send a participant's own audio packet out on the network is based both on information about the participant's own voice packet and voice packets that the participant receives from other clients. A Voice Activity Score (VAS) is computed for each frame of a particular voice stream. The VAS includes a voiceness component, indicating the likelihood that the audio frame contains speech or voice, and an energy level component that indicating the ratio of current frame energy to the long-term average of energy for a current speaker. Using the VAS from the participants, the method also ranks the client's voice stream as compared to other clients' voice streams in the voice conference. If there are participants higher ranking, the client's voice stream is not sent.