An image processing apparatus includes: a subject information storage unit configured to store feature quantities and attributes relating to a plurality of subjects; a subject detecting unit configured to detect a subject included in an image; an attribute determining unit configured to determine the attributes of the detected subject; a feature quantity extracting unit configured to extract a feature quantity relating to the detected subject; and a similarity calculating unit configured to select one or a plurality of feature quantities from feature quantities relating to a plurality of subjects stored in the subject information storage unit based on the determined attributes to calculate similarity between a subject according to the selected feature quantities, and the detected subject, based on the selected feature quantities and the extracted feature quantity.