Techniques for improving machine learning and text classification are described. The described techniques include improved processes for collecting training data to train a machine classifier. Some data sets are very large but contain only a small number of positive or negative training examples. The described text classification system obtains training examples by intelligently identifying documents that are likely to present or identify positive or negative training examples. The text classification system employs these techniques to train a classifier to categorize patent claims according some legal rule, such as subject-matter eligibility requirements under patent law.