System and method for estimating and tracking an orientation of a user's face by combining head tracking and face detection techniques. The orientation of the face, or facial pose, can be expressed in terms of pitch, roll and yaw of the user's head. Facial pose information can be used, for example, to ascertain in which direction the user is looking. In general, the facial pose estimation method obtains a position of the head and a position of the face and compares the two to obtain the facial pose. In particular, a camera is used to obtain an image containing a user's head. Any movement of the user's head is tracked and the head position is determined. A face then is detected on the head and the face position is determined. The head and face positions then are compared.