Embodiments of a system and method of controlling the loudness of an audio signal are disclosed. The method includes determining a long term loudness level for the audio signal over a first sliding time period, determining a short term loudness level for the audio signal over a second sliding time period where the second sliding time period is shorter than the first sliding time period, detecting a segment transition in the audio signal; and generating a gain control signal. The gain control signal is based on the short term loudness level for a first time interval after a segment transition and the gain control signal is based on the long term loudness level after the expiration of the first time interval. Other embodiments include addition signal analysis steps to identify different segments of a signal and to control the loudness of the segments.