A computer-implemented method, computer program product, and system are provided for learning mapping information between different modalities of data. The method includes mapping, by a processor, high-dimensional modalities of data into a low-dimensional manifold to obtain therefor respective low-dimensional embeddings through at least a part of a first network. The method further includes projecting, by the processor, each of the respective low-dimensional embeddings to a common latent space to obtain therefor a respective one of separate latent space distributions in the common latent space through at least a part of a second network. The method also includes optimizing, by the processor, parameters of each of the networks by minimizing a distance between the separate latent space distributions in the common latent space using a variational lower bound. The method additionally includes outputting, by the processor, the parameters as the mapping information.