A photographic subject tracking device includes: a first degree-of-similarity calculation unit that calculates degree of similarity between a template image for tracking and an image in search area; a photographic subject position identification unit that identifies a tracked photographic subject position in the input image based on calculated degree of similarity; a second degree-of-similarity calculation unit that calculates a degree of similarity between each of multiple template images for resizing determination, which are generated based on template image for tracking, and image in search area; a matching position identification unit that identifies matching positions of the multiple template images for resizing determination, respectively, in the input image based on calculated degrees of similarity; and a size changing unit that changes an image size of template image for tracking and template images for resizing determination based on a density of the plurality of matching positions identified.