A tracking device includes: a first tracking control unit that tracks an object based upon focus adjustment states that are detected by a focus detection unit in a plurality of focus detection positions; a second tracking control unit that tracks the object based upon image information that is outputted by an image-capturing unit and reference image information that has been set as a reference; a setting unit that sets a degree to which focus adjustment based upon a focus adjustment state detected by the focus detection unit is temporarily prohibited; and a control unit that selects one of the first tracking control unit and the second tracking control unit to be used for tracking the object, based upon the degree that has been set by the setting unit.