A method for classifying cases includes receiving a pool of unlabeled cases with associated per-case symmetrical importance scores, applying a selection algorithm with a classifier to a training set and the pool, but without the per-case symmetrical importance scores, to determine selection scores for the unlabeled case, and combining the selection scores and the corresponding per-case symmetrical importance scores to form combined scores for the unlabeled cases. The method further includes providing a high scoring unlabeled case to an oracle to label, receiving a labeled case back from the oracle and augmenting the training set with the labeled case, training the classifier with the augmented training set, and applying the classifier to an additional unlabeled case.