An autonomous vehicle is tested using virtual objects. The autonomous vehicle is maneuvered, by one or more computing devices, the autonomous vehicle in an autonomous driving mode. Sensor data is received corresponding to objects in the autonomous vehicle's environment, and virtual object data is received corresponding to a virtual object in the autonomous vehicle's environment. The virtual object represents a real object that is not in the vehicle's environment. The autonomous vehicle is maneuvered based on both the sensor data and the virtual object data. Information about the maneuvering of the vehicle based on both the sensor data and the virtual object data may be logged and analyzed.