In general, techniques are described by which to code scaled spatial components. A device comprising a memory and one or more processors may be configured to perform the techniques. The memory may store a bitstream including an encoded foreground audio signal and a corresponding quantized spatial component. The one or more processors may perform psychoacoustic audio decoding with respect to the encoded foreground audio signal to obtain a foreground audio signal, and determine, when performing psychoacoustic audio decoding, a bit allocation for the encoded foreground audio signal. The one or more processors may dequantize the quantized spatial component to obtain a scaled spatial component, and descale, based on the bit allocation, the scaled spatial component to obtain a spatial component. The one or more processors may reconstruct, based on the foreground audio signal and the spatial component, scene-based audio data.