An audio processing system for a listening device includes an input device, a voice activity detector and a ratio-based attenuator. The input device is configured to receive a first audio signal emanating from an environment and including a signal of interest. The voice activity detector is configured to generate a control signal in response to the first audio signal. The ratio-based attenuator is configured to receive the control signal and determine whether the signal level of the first audio signal exceeds the signal level of an audio signal received from an audio playback device by at least a target difference. If so, then the audio level of the playback audio signal is maintained. Otherwise, the audio level of the playback audio signal is adjusted, where, at the adjusted value, the first signal level exceeds the playback signal level by at least the target difference.