Mechanisms for automatically generating a set of test questions for use in generating a test to be administered to human test takers are provided. The mechanisms ingest a document from a corpus of documents to select a potential test question answer from the document and generate a candidate test question based on the potential test question answer. The mechanisms evaluate the candidate test question using a Question and Answer (QA) system to generate a result indicative of whether the candidate test question should be maintained for test generation. The mechanisms store the candidate test question in the set of test questions in response to a result indicating that the candidate test question should be maintained for test generation.