Systems and methods can quantify the tumor microenvironment for diagnosis, prognosis and therapeutic response prediction by fusing different data types (e.g., morphological information from histology and molecular information from omics) using an algorithm that harnesses deep learning. The algorithm employs tensor fusion to provide end-to-end multimodal fusion to model the pairwise interactions of features across multiple modalities (e.g., histology and molecular features) and deep learning. The systems and methods improve upon traditional methods for quantifying the tumor microenvironment that rely on concatenation of extracted features.