Systems and methods are provided for classifying an abnormality in a medical image. An input medical image depicting a lesion is received. The lesion is localized in the input medical image using a trained localization network to generate a localization map. The lesion is classified based on the input medical image and the localization map using a trained classification network. The classification of the lesion is output. The trained localization network and the trained classification network are jointly trained.