Disease detection from medical images is provided. In various embodiments, a medical image of a patient is read. The medical image is provided to a trained anatomy segmentation network. A feature map is received from the trained anatomy segmentation network. The feature map indicates the location of at least one feature within the medical image. The feature map is provided to a trained classification network. The trained classification network was pre-trained on a plurality of feature map outputs of the segmentation network. A disease detection is received from the trained classification network. The disease detection indicating the presence or absence of a predetermined disease.