Methods, systems, and apparatus, including computer-readable storage devices, for managing voice interface devices. In some implementations, messages are received from a plurality of devices, each of the messages indicating a respective voice input detected by the device that sent the message. Audio signatures are obtained for the voice inputs detected by the plurality of devices. The audio signatures for the voice inputs and times that the voice inputs were detected are evaluated. Based on the evaluation, at least some of the plurality of devices are grouped to form a group of multiple devices that detected a same user utterance. A device from the group is selected to respond to the user utterance, and the multiple devices in the group are managed so that only the selected device outputs a synthesized speech response to the user utterance.