A speech interface device may be configured to act as a remote speaker peripheral for multiple audio sources such as media players and phones. Upon receiving a request from a user to connect to an audio device, the speech interface device determines which of the multiple audio devices are currently available, selects one of the audio devices based on information about or received from the user, establishes an audio connection with the selected one of the audio devices, and begins acting as a remote speaker or speakerphone for the selected audio device.