A system and method track a face moving through multiple frames of a video sequence. A predicted position of a face in a video frame is obtained. A search is performed to determine appearance model similarity values of a detected face. Motion and appearance probabilities of the detected face are obtained, and the detected face in the video frame is associated with a track as a function of a joint probabilistic data association filter.