Systems and methods are disclosed herein for performing classification of documents or performing other tasks based on rules. A rule generator receives a request for a rule that will receive as an input a document and output an outcome such as a classification of the document, addition of the document to a whitelist or blacklist, or occurrence of some other outcome. The rules are applied to a document and the document and outcome of the rules are presented to a rater. A rating of the accuracy of the outcome is received from the rater and the rating is propagated to quality metrics of rules that contributed to the outcome. Rules with a quality metric above a threshold may be added to a production rule set. Rules with a quality metric below a threshold may be removed.