This disclosure describes methods, non-transitory computer readable storage media, and systems that utilize weakly supervised graph matching to align an ungrounded label graph and a visual graph corresponding to a digital image. Specifically, the disclosed system utilizes a label embedding model to generate label graph embeddings from the ungrounded label graph and a visual embedding network to generate visual graph embeddings from the visual graph. Additionally, the disclosed system determines similarity metrics indicating the similarity of pairs of label graph embeddings and visual graph embeddings. The disclosed system then generates a semantic scene graph by utilizing a graph matching algorithm to align the ungrounded label graph and the visual graph based on the similarity metrics. In some embodiments, the disclosed system utilizes contrastive learning to modify the embedding models. Furthermore, in additional embodiments, the disclosed system utilizes the semantic scene graph to train a scene graph generation neural network.