An electronic device may be provided with a color sensing ambient light sensor. The color sensing ambient light sensor may measure the color of ambient light. Control circuitry in the electronic device may use information from the color sensing ambient light sensor in adjusting a display in the electronic device or taking other action. The color sensing ambient light sensor may have light detectors with different spectral responses. A test system may be used to calibrate the color sensing light sensor. The test system may have a tunable light source with light-emitting diodes that are turned on in sequence while gathering measured responses from the detectors. Numerical optimization techniques may be used to produce final versions of the spectral responses for the light detectors from the measured responses and corresponding calibration data that is stored in the electronic device.