A system and method are provided for associating faces to determine whether the faces are similar. For example a captured face may be associated with known faces to determine which known face the captured most closely resembles. The system and method incorporate the use of Gabor filters applied to masked face images to develop covariance matrixes. The covariance matrixes are subject to a similarity measure to determine the similarity of the images.