An unsupervised acoustic modeling service for speech recognition is disclosed. A computing device may be present in a listening zone, such as a household or office, and may receive an audio signal that may include speech. In some instances, the speech is not directed to the computing device. Speech recognition results may be generated from the audio signal, using an acoustic model, and the results used to update the acoustic model. For example, the acoustic model may be updated to reflect a particular speaker's pronunciation of certain sound units, such as phonemes. By using speech that is not necessarily directed to the computing device, more data may be available for updating an acoustic model.