Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying a direction in which a user is looking. In one aspect, a method includes receiving an image of a sequence of images. The image can depict a face of a user. A template image for each particular facial feature point can be compared to one or more image portions of the image. The template image for the particular facial feature point can include a portion of a previous image of the sequence of images that depicted the facial feature point. Based on the comparison, a matching image portion of the image that matches the template image for the particular facial feature point is identified. A location of the matching image portion is identified in the image. A direction in which the user is looking is determined based on the identified location for each template image.