Mechanisms are provided for configuring a ranker array of a question and answer (QA) system. An initial configuration of a ranker array of the QA system is generated which comprises a plurality of rankers with each ranker applying a statistical classification function to candidate answers generated by the QA system. A training question is input to the QA system to generate a training candidate answer set and the ranker array is applied to the training candidate answer set to generate, for each ranker in the ranker array, a ranked listing of candidate answers. For each ranker in the ranker array, a correctness of a corresponding ranked listing of candidate answers is determined and the ranker array is reconfigured based on results of the determined correctness of the corresponding ranked listing of candidate answers for each ranker in the ranker array.