Data representative of the eye obtained from multiple modalities are combined into a virtual model of the eye. The multiple modalities indicate anatomical, physiological, and/or functional features of the eye. The data from different modalities is registered in order to combine the data into the virtual model. Eye data can be analyzed in the form of the virtual model. Neural networks can be used to obtain insights about medical conditions of the eye, such as for diagnosis or prognosis of conditions. It can also be predicted how the eye will respond to certain treatments.