Systems and methods for training a model for identifying an imaging modality. The systems and methods can be performed by a computer system having one or more processors and memory. A plurality of image vectors can be generated from first image data using a convolutional neural network. A loss function can be applied to each of the plurality of image vectors to produce an intermediate dataset. The intermediate dataset can be projected in a space having lower dimensional space that the intermediate dataset. A plurality of clusters can be identified from the intermediate dataset in the space using a clustering technique. Each of the plurality of clusters can be classified into one of a plurality of imaging modalities.