In an object tracking device, a search region setting unit sets the search region of an object in a frame image at a present point in time, based on an object region in a frame image at a previous point in time, zoom center coordinates in the frame image at the previous point in time, and a ratio between the zoom scaling factor of the frame image at the previous point in time and the zoom scaling factor of the frame image at the present point in time. A normalizing unit normalizes the image of a search region of the object included in the frame image at the present point in time to a fixed size. A matching unit searches the normalized mage of the search region for an object region similar to a template image.