Machine logic that automatically detects natural language processing (NLP) system annotation errors and correspondingly updates NLP annotators to prevent future erroneous annotations by performing the following steps: (i) determining that a first annotation error has occurred in an annotation of a corpus by the natural language processing system; (ii) generating a candidate set of annotation correction actions, where each annotation correction action of the set is adapted to prevent an occurrence of an error similar to the first annotation error by the natural language processing system; (iii) selecting an annotation correction action from the candidate set of annotation correction actions, based, at least in part, on a set of annotation correction confidence characteristics; and (iv) automatically applying the selected annotation correction action to the natural language processing system.