Systems and methods for the annotation of source data in accordance with embodiments of the invention are disclosed. In one embodiment, a data annotation server system obtains a set of source data, provides at least one subset of source data to at least one annotator device, obtains a set of annotation data from the at least one annotator device for each subset of source data, classifies the source data based on the annotation data using a machine classifier for each subset of source data, generates annotator model data describing the characteristics of the at least one annotator device, and generates source data model data describing at least one piece of source data in the set of source data, where the source data model data includes label data identifying the estimated ground truth for each piece of source data in the set of source data.