Loudness control is performed by estimating the energy and loudness of an audio signal. Loudness of the audio signal is determined by decomposing the audio signal into multiple frequency bands at different center frequencies. The energy of the audio signal in each frequency band is calculated and converted to a loudness, using a function that models human loudness perceptions of audio energies at different frequencies. The loudnesses are summed to obtain the total loudness of the audio signal. A signal gain is calculated as a function of a loudness setting provided by a human listener, the estimated signal energy, and the total loudness of the audio signal.