Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for query categorization based on image results. In one aspect, a method includes receiving images from image results responsive to a query, wherein each of the images is associated with an order in the image results and respective user behavior data for the image as a search result for the first query, and associating one or more of the first images with a plurality of annotations based on analysis of the selected first images' content.