Aspects of the disclosure provide for selecting viewports. A viewport is a sub-portion of an image. For example, a request to select a viewport for a particular image may be received. A set of potential viewports are identified within the particular image, and features are identified within the viewports of the set of potential viewports. A score is calculated for each given viewport of the set of potential viewports based at least in part on a weight vector having a set of feature weights and the identified features within that given viewport of the set of potential viewports. A viewport of the set of potential viewports is selected based on the calculated scores. The selected viewport is then provided in response to the request to select the viewport for the particular image.