The disclosure describes a speech detection system for detecting one or more desired speech segments in an audio stream. The speech detection system includes an audio stream input and a speech detection technique. The speech detection technique may be performed in various ways, such as using pattern matching and/or signal processing. The pattern matching implementation may extract features representing types of sounds as in phrases, words, syllables, phonemes and so on. The signal processing implementation may extract spectrally-localized frequency-based features, amplitude-based features, and combinations of the frequency-based and amplitude-based features. Metrics may be obtained and used to determine a desired word in the audio stream. In addition, a keypad stream having keypad entries may be used in determining the desired word.