A system and method for using images captured from a digital camera to control navigation through a three-dimensional user interface. The sequence of images may be examined to identify feature points to be tracked through successive frames of the images captured by the camera. A plurality of classifiers may be used to discern shift from rotation gestures, based on expected behavior of feature points in the image when the camera is shifted or rotated in position. The various classifiers may generate voting values for shift and rotation gestures, and the system can use historical gesture information to assist in categorizing a current gesture.