A residual scaling unit is configured to determine a scaling factor for a residual channel based on an inter-channel mismatch value. The inter-channel mismatch value is indicative of a temporal misalignment between a reference channel and a target channel. The residual scaling unit is further configured to scale (e.g., attenuate) the residual channel by the scaling factor to generate a scaled residual channel. A residual channel encoder is configured to encode the scaled residual channel as part of a bitstream.