An object tracking device stores, for each site of an object, information indicating an area image of the object in a first frame image of frame images included in a moving image, and stores a conspicuous site of the object. In response to the object being lost, for each maximum value position in a conspicuity map for a second frame image with the lost object, a recovery unit in the object tracking device (1) estimates an area with each site of the object when the conspicuous site is hypothetically at the maximum value position and (2) calculates a similarity score based on a similarity between the area image of each site in the first frame image and the area image of each site in the second frame image, and determines that the object is at a maximum value position in the second frame image with a maximum similarity score.