Methods, systems, and apparatus, including computer programs encoded on computer storage media, for selecting training images. One of the methods includes determining, for each of a plurality of labels that each designate a respective food class of a plurality of food classes, a respective measure of importance. A respective sample size is determined for the label based on the respective measure of importance of the label. A number of training images are selected for each respective label according to the determined sample size for the label. A predictive model is trained using the selected training images as training data.