An electronic device may be provided with a display mounted in a housing. An ambient light sensor may measure the color of ambient light through an ambient light sensor window in the display. The ambient light sensor may have an array of light detectors on a semiconductor die. The light detectors may include color matching function light detectors that have spectral sensitivity profiles that match standard observer color matching functions and may include spectral sensing light detectors. The spectral sensing light detectors may have narrower full width at half maximum bandwidths than the color matching function light detectors and may be used with the color matching function light detectors to measure an ambient light spectrum. The color matching function light detectors may be used to measure the color of ambient light.