An explainable artificially intelligent (XAI) application contains an ordered sequence of artificially intelligent software modules. When an input dataset is submitted to the application, each module generates an output dataset and an explanation that represents, as a set of Boolean expressions, reasoning by which each output element was chosen. If any pair of explanations are determined to be semantically inconsistent, and if this determination is confirmed by further determining that an apparent inconsistency was not a correct response to an unexpected characteristic of the input dataset, nonzero inconsistency scores are assigned to inconsistent elements of the pair of explanations. If the application's overall inconsistency score exceeds a threshold value, the system forwards information about the explanation, the offending modules, and the input dataset to a downstream machine-learning component that uses this information to train the application to better respond to future input that shares certain characteristics with the current input.