A speech enhancement system controls the gain of an excitation signal to prevent uncontrolled gain adjustments. The system includes a first device that converts sound waves into operational signals. An ambient noise estimator is linked to the first device and an echo canceller. The ambient noise estimator estimates how loud a background noise would be near the first device before or after an echo cancellation. The system then compares the ambient noise estimate to a current ambient noise estimate near the first device to control a gain of an excitation signal.