Embodiments of the present systems and methods may provide the capability to classify medical images, such as mammograms, in an automated manner using existing annotation information. In embodiments, only the global, image level tag may be needed to classify a mammogram into certain types, without fine annotation of the findings in the image. In an embodiment, a computer-implemented method for classifying medical images may comprise receiving a plurality of image tiles, each image tile including a portion of a whole view, processed by a trained or a pre-trained model and outputting a one-dimensional feature vector for each tile to generate a three-dimensional feature volume and classifying the larger image by a trained model based on the generated three-dimensional feature volume to form a classification of the image.