An approach is provided for evaluating subject matter experts (SMEs) in a question and answering (QA) system. In the approach, a number of responses are received at the QA system with each of the responses being a response to a common question and each of the responses being received from a SME. One of the responses is selected with the selected response being from one of the SMEs that is being evaluated. The approach evaluates the selected response by comparing the selected response to the responses received from the other SMEs. Based on the evaluation, the approach updates a SME confidence score that corresponds to the selected SME.