An autonomous vehicle (AV) described herein is configured to receive a pull over location specified by a passenger, and is further configured to refine the pull over location based upon one or more factors, where the factors include computer-readable content from a profile of the passenger, sensor data output by sensor systems of the AV, observed or predicted weather conditions, observed or predicted traffic, and/or observations recently generated by other AVs that belong to the same fleet as the AV.