In general, techniques are described for closed loop quantization of HOA coefficients that provide a three-dimensional representation of the sound field. An audio encoding device may perform closed loop quantization of an audio object based at least in part on a result of performing quantization of directional information associated with the audio object. An audio decoding device may obtain an audio object that has been closed loop quantized based at least in part on a result of performing quantization of directional information associated with the audio object, and may dequantize the audio object.