In general, techniques are described by which to perform audio coding based on audio pattern recognition. A source device comprising a memory and a processor may be configured to perform the techniques. The memory may store audio data. The processor may obtain, from a plurality of categories, a category to which the audio data corresponds, and obtain, based on the category, a set of pyramid vector quantization (PVQ) parameters from a plurality of sets of PVQ parameters. The processor may also perform, based on the set of PVQ parameters, PVQ with respect to the audio data to obtain a residual identifier representative of the audio data, and specify, in the bitstream, the residual identifier.