Personal audio systems and methods are disclosed. A personal audio system includes a voice activity detector to determine whether or not an ambient audio stream contains voice activity, a pitch estimator to determine a frequency of a fundamental component of an annoyance noise contained in the ambient audio stream, and a filter bank to attenuate the fundamental component and at least one harmonic component of the annoyance noise to generate a personal audio stream. The filter bank implements a first filter function when the ambient audio stream does not contain voice activity, or a second filter function when the ambient audio stream contains voice activity.