Various embodiments of a system and methods for using contextual features to improve face recognition in digital images are described. A face recognition system may semi-automatically label faces and/or search for faces in images. Faces may be detected in the images and visual and non-visual features may be determined for each face. The determined features may be facial and/or contextual features. Features may also be determined from labels assigned to the faces. Pair-wise comparisons of the faces, using the determined features, may determine distances between pairs of faces. Some features may be used to determine the relevance of other features. Dependent on the calculated distances, similar faces may be grouped and suggested labels for faces may be provided. The system may be scalable, processing faces in groups to limit the use of system resources to a certain amount, regardless of the size of a digital image collection.