A mechanism is provided in a data processing system for determining comprehensiveness of a question paper given a syllabus of topics. An answer and evidence generator of a question answering system executing on the data processing system finds one or more answers based on the syllabus of topics for each question in the question paper. The answer and evidence generator identifies evidence for the one or more answers in the syllabus for each question in the question paper. A concept identifier of the question answering system identifies a set of concepts in the syllabus corresponding to the evidence for each question in the question paper to form a plurality of sets of concepts. The mechanism determines a value for a comprehensiveness metric for the question paper with respect to the syllabus of topics based on the plurality of sets of concepts.