A subject tracking device includes: a first similarity factor calculation unit that compares an input image assuming characteristics quantities corresponding to a plurality of characteristics components, with a template image assuming characteristics quantities corresponding to the plurality of characteristics components, and calculates a similarity factor indicating a level of similarity between the input image and the template image in correspondence to each of the plurality of characteristics components; a normalization unit that normalizes similarity factors corresponding to the plurality of characteristics components having been calculated by the first similarity factor calculation unit; and a second similarity factor calculation unit that calculates a similarity factor indicating a level of similarity between the input image and the template image based upon results of normalization achieved via the normalization unit.