In one embodiment, a method includes identifying, based on map data, an area for pick-up or drop-off of a user by an autonomous vehicle. The method also includes determining, based on autonomous-vehicle sensor data that represents an external environment of the autonomous vehicle, one or more potential pick-up or drop-off locations within the area. The method also includes calculating, based at least in part on the autonomous-vehicle sensor data and historical data a viability score for each of the potential pick-up or drop-off locations. The historical data includes past pick-up or drop-off locations for one or more past users. The method also includes providing for display a visual representation of at least a portion of the area for pick-up or drop-off that indicates at least one of the one or more potential pick-up or drop-off locations.