A feature tracking technique for detecting and tracking feature points with primary colors. An energy value may be computed for each color channel of a feature. If the energy of all the channels is above a threshold, then the feature may be tracked according to a feature tracking method using all channels. Otherwise, if the energy of all of the channels is below the threshold, then the feature is not tracked. If the energy of at least one (but not all) of the channels is below the threshold, then the feature is considered to have primary color, and the feature may be tracked according to the feature tracking method using only the one or more channels with energy above the threshold. The feature tracking techniques may, for example, be used to establish point trajectories in an image sequence for various Structure from Motion (SFM) techniques.