An audio decoder for decoding an encoded audio signal, the encoded audio signal including a first channel element and a second channel element in a payload section of a data stream and first decoder configuration data for the first channel element and second decoder configuration data for the second channel element in a configuration section of the data stream, includes: a data stream reader for reading the configuration data for each channel element in the configuration section and for reading the payload data for each channel element in the payload section; a configurable decoder for decoding the plurality of channel elements; and a configuration controller for configuring the configurable decoder so that the configurable decoder is configured in accordance with the first decoder configuration data when decoding the first channel element and in accordance with the second decoder configuration data when decoding the second channel element.