The disclosed embodiments provide a system that generates sound primitives to facilitate sound recognition. First, the system performs a feature-detection operation on sound samples to detect a set of sound features, wherein each sound feature comprises a measurable characteristic of a window of consecutive sound samples. Next, the system creates feature vectors from coefficients generated by the feature-detection operation, wherein each feature vector comprises a set of coefficients for sound features detected in a window. The system then performs a clustering operation on the feature vectors to produce feature-vector clusters, wherein each feature-vector cluster comprises a set of feature vectors that are proximate to each other in a feature-vector space that contains the feature vectors. After the clustering operation, the system defines a set of sound primitives, wherein each sound primitive is associated with a feature-vector cluster. Finally, the system associates semantic labels with the set of sound primitives.