A color ambient light sensor may gather ambient light measurements during operation of an electronic device. The color ambient light sensor may have a color ambient light detector and an adjustable neutral density filter. The electronic device may have components such as a camera and display that are adjusted using ambient light information from the color ambient light sensor. The display may have a display cover layer. Pixels in an active area of the display may display images through the display cover layer. An inactive area of the display may have an opaque masking layer on an interior surface of the display cover layer. An opening in the opaque masking layer may form an ambient light sensor window for the color ambient light sensor. The adjustable neutral density filter may be interposed between the color ambient light detector and the ambient light sensor window.