Presented herein are techniques for generating a hierarchical classification of a set of sound signals received at hearing prosthesis. The hierarchical classification includes a plurality of nested classifications of a sound environment associated with the set of sound signals received at hearing prosthesis, including a primary classification and one or more secondary classifications that each represent different characteristics of the sound environment. The primary classification represents a basic categorization of the sound environment, while the secondary classifications define sub-categories/refinements of the associated primary classification and/or other secondary classifications.