A system and method, which may be an offline method, extracts relevant image features about images in a network-based publication system for enabling image similarity searching of such images. An image is uploaded and may be sent to a picture processing service, which generates digests. The digests are compressed data structures each representing a particular image feature such as edge, color, texture, or words. These digests are then stored in a search database, where the digests can be used to retrieve images by image similarity at scale. A similar process can be performed for an image query for searching the search database for images similar to the image.