An electronic device may be provided with an ambient light sensor, a display that displays image content, and control circuitry. The control circuitry may adjust a peak allowable brightness of the display based on an ambient light brightness and based on the image content being displayed. For example, the control circuitry may analyze frames of display data to determine an average pixel luminance level. Low average pixel luminance levels correspond to mostly dark image content, whereas high average pixel luminance levels correspond to mostly light image content. When an electronic device is outdoors and displaying mostly dark images with low average pixel luminance levels, the control circuitry may take advantage of the display's maximum achievable brightness to improve readability. When an electronic device is outdoors and displaying mostly light images with high average pixel luminance levels, the control circuitry may scale the maximum allowable brightness down to reduce power consumption.