Automatically determining a medical recommendation for a patient based on multiple medical images from multiple different medical imaging modalities. In some embodiments, a method may include receiving a first and second medical images of a patient from first and second medical imaging modalities, mapping a first region of interest (ROI) on the first medical image to a second ROI on the second medical image, generating first annotation data related to the first ROI and second annotation data related to the second ROI, generating first medical clinical data related to the first ROI and second medical clinical data related to the second ROI, inputting, into a machine learning classifier, the first and second annotation data and the first and second medical clinical data, and automatically determining, by the machine learning classifier, a medical recommendation for the patient related to a medical condition of the patient.