Systems and techniques for classification based on annotation information are presented. In one example, a system trains a convolutional neural network based on training data and a plurality of images. The training data is associated with a plurality of patients from at least one imaging device. The plurality of images is associated with a plurality of masks from a plurality of objects. The system also generates a loss function based on the plurality of masks, where the loss function is iteratively back propagated to tune parameters of the convolutional neural network. The system also predicts a classification label for an input image based on the convolutional neural network.