The present invention is a method and system to provide a face-based automatic ethnicity recognition system that utilizes ethnicity-sensitive image features and probabilistic graphical models to represent ethnic classes. The ethnicity-sensitive image features are derived from groups of image features so that each grouping of the image features contributes to more accurate recognition of the ethnic class. The ethnicity-sensitive image features can be derived from image filters that are matched to different colors, sizes, and shapes of facial features—such as eyes, mouth, or complexion. The ethnicity-sensitive image features serve as observable quantities in the ethnic class-dependent probabilistic graphical models, where each probabilistic graphical model represents one ethnic class. A given input facial image is corrected for pose and lighting, and ethnicity-sensitive image features are extracted. The extracted image features are fed to the ethnicity-dependent probabilistic graphical models to determine the ethnic class of the input facial image.