A mechanism is provided in a data processing system for conducting error analysis for a question answering system. Responsive to the question answering system generating one or more candidate answers for an input question, wherein the one or more candidate answers are determined to be incorrect, the mechanism instantiates a plurality of instances of the question answering system with a modification to each instance. The mechanism provides the input question to each of the plurality of instances of the question answering system. The mechanism analyzes results from the plurality of instances of the question answering system to identify at least one modification that led to improved results. The mechanism presents a graphical output based on the analysis.