Aspects of the disclosure relate to detecting and responding to stop signs. An object detected in a vehicle's environment having location coordinates may be identified as a stop sign and, it may be determined whether the location coordinates of the identified stop sign correspond to a location of a stop sign in detailed map information. Then, whether the identified stop sign applies to the vehicle may be determined based on the detailed map information or on a number of factors. Then, if the identified stop sign is determined to apply to the vehicle, responses of the vehicle to the stop sign may be determined, and, the vehicle may be controlled based on the determined responses.