An apparatus and method for tracking an object using a feature descriptor and an apparatus and method for removing a garbage feature are disclosed. A feature descriptor generation unit generates a plurality of features descriptors indicating information of a plurality of features extracted from an input image from which an object of interest is desired to be detected. A matching unit matches the feature descriptors with feature descriptors of a target object stored in advance, and determines the feature descriptors of the object of interest corresponding to the target object. A feature point removal unit removes feature descriptors that do not meet a geometric comparison condition from among the feature descriptors of the object of interest, and establishes final feature descriptors of the object of interest.