Mechanisms are provided to implement a neural network, a concept extractor, and a machine learning model that operate to provide automatic contour annotation of medical images based on correlations with medical reports. The neural network processes a medical image to extract image features of the medical image. The concept extractor processes a portion of text associated with the medical image to extract concepts associated with the portion of text. The machine learning model correlates the extracted image features with the extracted concepts. An annotated medical image is generated based on the correlation of the extracted image features and extracted concepts. An annotation of the annotated medical image specifies a region of interest corresponding to both an extracted image feature and an extracted concept, thereby automatically mapping the portion of text to a relevant region of the medical image.