Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for adjusting a brightness of a display based on an image. In one aspect, a method includes obtaining an image, determining an amount of brightness in the image, determining an amount of contrast in the image, determining a brightness gain that reflects an amount to adjust brightness of a display based on both the amount of brightness in the image and the amount of contrast in the image, adjusting the brightness of the display based on the brightness gain, and providing the image for output on the display with the brightness of the display adjusted in accordance with the brightness gain.