Techniques performed by a data processing system for analyzing the impact of training data changes on a machine learning model herein include training a first instance of a machine learning model with a first set of training data; modifying the first set of training data to produce a second set of training data; training a second instance of the model with the second set of training data; comparing the first instance of the model to the second instance of the model to determine features that differ between the first instance and the second instance of the model; identifying a subset of historical data associated with the features that differ between the first instance and the second instance of the model; and scoring the subset of the historical data to produce a report identifying differences in the output of the first instance and the second instance of the machine learning model.