Systems and techniques are provided for a ranking approach to train deep neural nets for multilabel image annotation. Label scores may be received for labels determined by a neural network for training examples. Each label may be a positive label or a negative label for the training example. An error of the neural network may be determined based on a comparison, for each of the training examples, of the label scores for positive labels and negative labels for the training example and a semantic distance between each positive label and each negative label for the training example. Updated weights may be determined for the neural network based on a gradient of the determined error of the neural network. The updated weights may be applied to the neural network to train the neural network.