The technology disclosed extends Human-in-the-loop (HITL) active learning to incorporate real-time human feedback to influence future sampling priority for choosing the best instances to annotate for accelerated convergence to model optima. The technology disclosed enables the user to communicate with the model that generates machine annotations for unannotated instances. The technology disclosed also enables the user to communicate with the sampling logic that selects instances to be annotated next. The technology disclosed enables the user to generate ground truth annotations, from scratch or by correcting erroneous model annotations, which guide future model predictions to more accurate results. The technology disclosed enables the user to optimize the sampling logic to increase the future sampling likelihood of those instances that are similar to the instances that the user believes are informative, and decrease the future sampling likelihood of those instances that are similar to the instances that the user believes are non-informative.