The technology relates to detecting and responding to sounds for a vehicle having an autonomous driving mode. In one example, an audible signal corresponding to a sound received at one or more microphones of the vehicle may be received. Sensor data generated by a perception system of the vehicle identifying objects in an environment of the vehicle may be received. A type of sound may be determined by inputting the audible signal into a classifier. A set of additional signals may be determined based on the determined type of sound. The sensor data may be processed in order to identify one or more additional signals of the identified set of additional signals. The vehicle may be controlled in the autonomous driving mode in order to respond to the sound based on the one or more additional signals and the type of sound.