Method and apparatus for image retrieval. An image descriptor of a query image indicating a medical abnormality and non-image patient data may be received. For each of a plurality of candidate images stored in a database, an image descriptor of the candidate image and data about a medical abnormality known to be indicated by the candidate image may further be received. A similarity metric between the image descriptors of the query image and the candidate image may be determined for each candidate image. A first probability of the query image medical abnormality being the candidate image medical abnormality given the non-image patient data associated with the query image may be determined for each candidate image. A score may then be determined for each candidate image based on the determined similarity metric and the determined first probability. One or more of the candidate images may be retrieved from the database in accordance with the determined scores.