A method for providing adaptation to a question answering (QA) system having an associated plurality of trace data and a question-answer set. The method includes submitting a set of questions to the QA system; receiving back from the QA system a set of answers; comparing the set of answers to answers in the question-answer set; and generating the plurality of trace data based on comparison, and an estimate of how much more training data is needed. The generating comprises (a) for each question type, successively sample an increasing number of question-answer pairs from the question-answer set; (b) automatically train the QA system using the sampled question-answer pairs; (c) automatically compute a functional dependence of the QA system performance on the remaining questions relative to the size of the sample; and (d) extrapolate from the functional dependence a number of training question-answer pairs of each question type.