In a similar face retrieval system for retrieving an image photographing a face similar to a face detected from a retrieval query image from a retrieval target image group by using an image photographing a human face as the retrieval query image, whole image features as features representative of background information are extracted from each whole area of an each image of a retrieval target image group, to calculate a degree of similarity through comparison with each set of whole image features, and an image having a degree of similarity not lower than a certain value and having a lower retrieval result order from retrieval results. It is possible to efficiently retrieve the same person playing in different scenes by utilizing different features for a retrieval process and a filtering process.