A system and method are provided for detecting one or both eyes of the driver of a vehicle. The system includes a video imaging camera oriented to generate images of the driver's face. The system also includes a video processor for processing the images generated with the video imaging camera. Filters are applied to each of potential eye candidates to determine which candidates represent an eye, weighted values are assigned to the filtered eye candidates, and an eye is detected based on the weighting of each eye candidate. According to one aspect, different size patches of potential eye candidates are processed and compared to models.