In one embodiment, a computing system of a vehicle generates perception data based on sensor data captured by one or more sensors of the vehicle. The perception data includes one or more representations of physical objects in an environment associated with the vehicle. The computing system further determines simulated perception data that includes one or more representations of virtual objects within the environment and generates modified perception data based on the perception data and the simulated perception data. The modified perception data includes at least one of the one or more representations of physical objects and the one or more representations of virtual objects. The computing system further determines a path of travel for the vehicle based on the modified perception data, which includes the one or more representations of the virtual objects.