Systems and methods for assessing a disease are provided. An input medical image in a first modality is received. Lungs are segmented from the input medical image using a trained lung segmentation network and abnormality patterns associated with the disease are segmented from the input medical image using a trained abnormality pattern segmentation network. The trained lung segmentation network and the trained abnormality pattern segmentation network are trained based on 1) synthesized images in the first modality generated from training images in a second modality and 2) target segmentation masks for the synthesized images generated from training segmentation masks for the training images. An assessment of the disease is determined based on the segmented lungs and the segmented abnormality patterns.