Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a final classification output for an image of eye tissue. The image is provided as input to each of one or more segmentation neural networks to obtain one or more segmentation maps of the eye tissue in the image. A respective classification input is generated from each of the segmentation maps. For each of the segmentation maps, the classification input for the segmentation map is provided as input to each of one or more classification neural networks to obtain, for each segmentation map, a respective classification output from each classification neural network. A final classification output for the image is generated from the respective classification outputs for each of the segmentation maps.