A tracking apparatus includes a grouping setting unit, a tracking feature detection unit, a tracking unit. The grouping setting unit groups a plurality of focus detection areas with an in-focus state. The tracking feature detection unit detects a feature amount of the tracking target in areas of the groups grouped. The tracking unit tracks the tracking target in accordance with a first or second tracking position depending on the number of the set groups.