Methods, systems, and apparatus, including computer program products, for evaluating image data. In one aspect, a method includes accessing an image that includes a candidate face, such as a face detected during a face detection operation. The method further includes generating a sharpness measure based on image data corresponding to the candidate face, evaluating the sharpness measure to determine a confidence score representing a likelihood that the candidate face corresponds to a human face, and accepting the candidate face when the confidence score compares in a predetermined manner to a confidence threshold. Additionally, the method can be implemented to include generating a skin tone measure based on image data corresponding to the candidate face and evaluating the sharpness measure in combination with the skin tone measure to determine the confidence score.