Mechanisms are provided for modifying an operation of a question answering (QA) system. An input question is received and processed to generate at least one query to be applied to a corpus of information. The at least one query is applied to the corpus of information to generate candidate answers to the input question from which a final answer is selected for output. A training engine modifies, using a machine learning technique that compares the final answer to a known correct answer for the input question, at least one of logic or configuration parameters of the QA system for at least one of the processing of the input question to generate the at least one query, applying of the at least one query to the corpus of information to generate the candidate answers, or the selecting of the final answer from the candidate answers.