A classifier training system trains unified classifiers for categorizing videos representing different categories of a category graph. The unified classifiers unify the outputs of a number of separate initial classifiers trained from disparate subsets of a training set of media items. The training process takes into account the relationships that exist between the various categories of the category graph by relating scores associated with related categories, thus enhancing the accuracy of the unified classifiers.