Described is automatically testing the quality of an audio channel between a caller and a callee that includes a device under test, such as a VoIP or other gateway. An analyzer receives timestamps from a caller and callee during a calling session, including timestamps for when the callee initially provides audio (e.g., speech) to the caller, when the caller initially detects sound, when the caller initially provides audio to the callee, and when the callee initially detects sound. The analyzer uses the relative timing of the timestamps and the speech recognizer's outcome to determine whether the audio channel is experiencing interference or echo. When the audio includes speech, a confidence level corresponding to accuracy of speech recognition also may establish the audio channel's quality. Random selection and timing of output may be employed, such as to vary the testing patterns during repetitive tests.