Techniques for simulating interactions using an augmented reality device are described. Embodiments receive a request to paint over portions of a visual scene. Here, the visual scene is captured using one or more camera devices of the augmented reality device and is presented on a display of the augmented reality device. A first object in the visual scene corresponding to the first location is identified. Additionally, embodiments paint at least a portion of the first object as specified by the received request, and render a series of frames depicting the first object, such that the painted at least a portion of the first object is shown as painted when viewed from different perspectives using the augmented reality device.