The subject matter of this specification can be embodied in, among other things, a computer-implemented method that includes receiving a plurality of images having human faces. The method further includes generating a data structure having representations of the faces and associations that link the representations based on similarities in appearance between the faces. The method further includes outputting a first gender value for a first representation of a first face that indicates a gender of the first face based on one or more other gender values of one or more other representations of one or more other faces that are linked to the first representation.