Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for searching within images. In one aspect, a method includes receiving a query from a user device; receiving a query image from the user device; identifying a result image responsive to the query; determining that an object depicted in a sub-portion of the query image is similar to the result image; and in response to determining that the object depicted in the sub-portion of the query image is similar to the result image: providing annotation data to the user device that causes the user device to display an annotation with the query image, the annotation specifying the sub-portion of the query image that includes the object that is similar to the result image.