In a face recognition system, overlapping patches are defined on a canonical face. Random clusters of pixel pairs are defined within each patch, and binary features are determined for each pixel pair by comparing their respective feature values. An inverted index hash table is constructed of the binary features. Similar binary features are then determined on a library of registrable samples of identified faces. A log probability of each registrable sample generating a binary feature from a corresponding cluster of pixel pairs at each specific patch location is determined and stored in the hash table. In a search phase, similar binary features are determined, and a hash key is determined for each binary feature. The log probabilities for each identity found in the hash table are summed for all clusters of pixel pairs and locations and sorted to find the high probability match.