A personal audio device that detects speech provides for improved interaction with others. When speech is detected in a microphone output signal of a microphone that measures ambient audio sounds, the audio program being reproduced by the personal audio device may be altered, by attenuating, muting or interrupting the program material. The speech may be provided to a headset that reproduces the program material. The direction of the speech can be used to determine whether the speech is from a person other than the use of the personal audio device.