Aspects of the disclosure provide systems and methods for confirming the identity of a passenger and changing destination of a vehicle. This may include receiving dispatching instructions to pick up a first passenger at a pickup location and to drop off the first passenger at a first destination as well as authentication information for authenticating a first client computing device of the first passenger. Once the client device is authenticated and a second passenger enters the vehicle, the vehicle is maneuvered towards the first destination. While doing so, a location of the vehicle is compared to location information received from the client computing devices. A notification is sent to a dispatching server based on the comparison and a second destination location is received in response. The vehicle is then maneuvered towards the second destination instead of the first destination.