Embodiments are directed towards classifying data using machine learning that may be incrementally refined based on expert input. Data provided to a deep learning model that may be trained based on a plurality of classifiers and sets of training data and/or testing data. If the number of classification errors exceeds a defined threshold classifiers may be modified based on data corresponding to observed classification errors. A fast learning model may be trained based on the modified classifiers, the data, and the data corresponding to the observed classification errors. And, another confidence value may be generated and associated with the classification of the data by the fast learning model. Report information may be generated based on a comparison result of the confidence value associated with the fast learning model and the confidence value associated with the deep learning model.