In one example, a headset obtains, from a first microphone on the headset, a first audio signal including a user audio signal and an anisotropic background audio signal. The headset obtains, from a second microphone on the headset, a second audio signal including the user audio signal and the anisotropic background audio signal. The headset extracts, from the first audio signal and the second audio signal, using a first adaptive filter, a reference audio signal including the anisotropic background audio signal. Based on the reference signal, the headset cancels, using a second adaptive filter, the anisotropic background audio signal from a third audio signal derived from the first and second audio signals to produce an output audio signal. The headset provides the output audio signal to a receiver device.