A multi-label heat map generating system is operable to receive a plurality of medical scans and a corresponding plurality of medical labels that each correspond to one of a set of abnormality classes. A computer vision model is generated by training on the medical scans and the medical labels. Probability matrix data, which includes a set of image patch probability values that each indicate a probability that a corresponding one of the set of abnormality classes is present in each of a set of image patches, is generated by performing an inference function that utilizes the computer vision model on a new medical scan. Preliminary heat map visualization data can be generated for transmission to a client device based on the probability matrix data. Heat map visualization data can be generated via a post-processing of the preliminary heat map visualization data to mitigate heat map artifacts.