In techniques for category histogram image representation, image segments of an input image are generated and bounding boxes are selected that each represent a region of the input image, where each of the bounding boxes include image segments of the input image. A saliency map of the input image can also be generated. A bounding box is applied as a query on an images database to determine database image regions that match the region of the input image represented by the bounding box. The query can be augmented based on saliency detection of the input image region that is represented by the bounding box, and a query result is a ranked list of the database image regions. A category histogram for the region of the input image is then generated based on category labels of each of the database image regions that match the input image region.