Imaging devices may include processing circuitry, a lens, and an array of image sensor pixels and reference pixels. The array may receive direct image light and stray light from the lens. The image sensor pixels may include clear color filter elements and the reference pixels may include opaque color filter elements. The opaque color filter elements may block direct image light from being captured by the reference pixels. The image sensor pixels may generate pixel values in response to the direct image light and the stray light whereas the reference pixels may generate reference pixel values in response to the stray light. The processing circuitry may mitigate stray light effects such as local flare and veiling glare within the imaging system by adjusting the pixel values based on the reference pixel values. The imaging system may be calibrated in a calibration system for generating stray light calibration data.