Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a clinical recommendation for medical treatment of a patient. In one aspect a method comprises: receiving multi-modal data characterizing a patient, wherein the multi-modal data comprises a respective feature representation for each of a plurality of modalities; processing the multi-modal data characterizing the patient using a machine learning model, in accordance with values of a set of machine learning model parameters, to generate a patient classification that classifies the patient as being included in a patient category from a set of patient categories; determining an uncertainty measure that characterizes an uncertainty of the patient classification generated by the machine learning model; and generating a clinical recommendation for medical treatment of the patient based on: (i) the patient classification, and (ii) the uncertainty measure that characterizes the uncertainty of the patient classification.