Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating image search results. One of the methods includes receiving first image search results responsive to a text query, each first image search result associated with a respective first score indicating a relevance of an image represented by the first image search result to the text query. Second image search results responsive to a query image are received, each second image search result associated with a respective second score indicating a measure of similarity between an image represented by the second image search result and the query image. A set of final image search results is selected including combining first scores and second scores of the selected first image search results. The final image search results are ordered by similarity to the query image.