Embodiments herein describe an AR system which communicates with light sources in a real world environment to simulate the light sources having an effect on AR content being displayed in the AR system. In one embodiment, the AR system determines the light characteristics of the light sources such as their color, specularity, beam spread, brightness, or illumination pattern. These light characteristics can contribute to the ambient light in the real world environment which affects the appearance of the physical objects in the environment. The AR system can adjust how the AR content is displayed to simulate the AR content being affected in a similar manner by the light sources. As such, the AR content appears to be more like a physical object (rather than a virtual object) thereby providing the user with a more immersive experience.