In processing a multi-channel audio signal having at least three original channels, a first downmix channel and a second downmix channel are provided, which are derived from the original channels. For a selected original channel, channel side information are calculated such that a downmix channel or a combined downmix channel including the first and the second downmix channels, when weighted using the channel side information, results in an approximation of the selected original channel. The channel side information and the first and second downmix channels form output data for transmission to a decoder. A low level decoder only decodes the first and second downmix channels. A high level decoder provides a full multi-channel audio signal based on the downmix channels and the channel side information. Since the channel side information only occupy a low number of bits and the decoder does not use dematrixing, efficient and high quality multi-channel extension for stereo players and enhanced multi-channel players is obtained.