An electronic device may be provided with a display mounted in a housing. A color sensing ambient light sensor may measure the color of ambient light. The color sensing ambient light sensor may be mounted in alignment with an ambient light sensor window formed in an inactive area of the display. The color sensing ambient light sensor may be formed from detectors on a semiconductor substrate. The detectors may include detectors that have spectral sensitivity profiles matching those of color matching functions. The color sensing ambient light sensor may include an infrared light detector. Light redirecting structures such as a diffuser, prism film, negative lens film, or privacy film may be used in directing light into the ambient light sensor. The color sensing ambient light sensor may be calibrated by exposing the sensor to light sources of different types.