Aspects of the disclosure provide for reducing inconvenience to other road users caused by stopped autonomous vehicles. As an example, a vehicle having an autonomous driving mode may be stopped at a first location. While the vehicle is stopped, sensor data is received from a perception system of the vehicle. The sensor data may identify a road user. Using the sensor data, a value indicative of a level of inconvenience to the road user caused by stopping the vehicle at the first location may be determined. The vehicle is controlled in the autonomous driving mode to cause the vehicle to move from the first location and in order to reduce the value.