Realistic reflections can be generated for augmented reality content displayed with respect to a view of a physical scene. A device executing an AR application obtains a three dimensional representation of the scene. The representation is used to determine surface orientation information for various physical objects in the physical scene, indicating how incident light would be reflected from those surfaces. The device can also use an infrared detector or transceiver to obtain an infrared view of the scene. The infrared data can be analyzed to determine the reflectivity of the various surfaces in the scene. Utilizing the surface orientation and reflectivity data, reflections from various AR content rendered for the scene can be determined. The corresponding pixels of the live view of the scene can then be color adjusted to account for the reflection, or an overlay can be generated that simulates the color adjustment for the reflection.