A system and method to encode and decode multiple audio signals to provide independent control of the audio signals is provided. A host device may encode the audio signals to enable a complete separation of the constituent audio signals when the mixed stream is decoded on a playback device. The gains of the audio signals may be independently controlled before they are mixed to increase the intelligibility of one audio signal relative to another audio signal at the playback device. The ability to separate the constituent audio signals from the mixed signals at the playback device allows the processing operations performed on the constituent audio signals and the associated path latencies to be independently chosen. In addition, in applications where the mixed stream is transmitted from a single host device to multiple playback devices, the constituent audio signals may be selectively masked on a playback device to increase user privacy.