A system and method are disclosed for learning a random multinomial logit (RML) classifier and applying the RML classifier for scene segmentation. The system includes an image textonization module, a feature selection module and a RML classifier. The image textonization module is configured to receive an image training set with the objects of the images being pre-labeled. The image textonization module is further configured to generate corresponding texton images from the image training set. The feature selection module is configured to randomly select one or more texture-layout features from the texton images. The RML classifier comprises multiple multinomial logistic regression models. The RML classifier is configured to learn each multinomial logistic regression model using the selected texture-layout features. The RML classifier is further configured to apply the learned regression models to an input image for scene segmentation.