A classification system classifies different aspects of content of an input image stream, such as faces, landmarks, events, and so forth. The classification system includes a general classifier and at least one specialized classifier template. The general classifier is trained to classify a large number of different aspects of content, and a specialized classifier can be trained based on a specialized classifier template during operation of the classification system to classify a particular subset of the multiple different aspects of content. The classification system determines when to use the general classifier and when to use a specialized classifier based on class skew, which refers to the temporal locality of a subset of aspects of content in the image stream.