An approach for responding to a text-based query for a digital image is provided. A request that identifies one or more keywords is received. A number of annotated digital images are selected. Each selected annotated digital image has a bounded region, on its appearance, that has an annotation associated with at least one of the keywords. A set of candidate digital images is selected for each annotated digital image. The set of candidate images, for a particular annotated digital image, are the digital images, of a set of digital images, which have an appearance that is most similar to the particular annotated digital image. The sets of candidate images are aggregated into a single set of digital images. A response is generated that identifies those digital images in the single set of digital images which are most responsive to the one or more keywords.