It is possible to shorten a delay time from utterance to output from a speaker even in a conference audio system including an automatic mute release device. An A/D converter that converts audio signals from a plurality of microphones into digital signals, an audio level detector that detects utterance or silence depending on the level of the converted digital signal, an audio data storage unit that temporarily stores the digital signal for which the audio level detector detected utterance, a controller that controls the storage of audio data in the audio data storage unit and the reading of the audio data, and a D/A converter that converts the read audio data into analog audio signals are provided. The controller hastens read timing of the audio data in accordance with a time period of silent portion when the audio level detector detects silence in a series of the audio data.