Photograph organization based on facial recognition is described. In one or more embodiments, a photograph organization module obtains multiple photographs having images of faces and recognizes the faces in the multiple photographs. The module builds a population by attempting to distinguish individual persons among the faces in the multiple photographs, with each person of the population corresponding to a group of multiple groups. After a first pass through the faces, the population includes a number of duplicative persons. With a second pass, the photograph organization module reduces the number of duplicative persons in the population by merging two or more groups of the multiple groups to produce a reduced number of groups. The merging is performed based on comparisons of the faces corresponding to the two or more groups. The multiple photographs are organized based on the reduced number of groups. Organization can include tagging or displaying grouped photographs.