Among other things, we describe techniques for implementing a vehicle response to sensor failure. In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include receiving information from a plurality of sensors coupled to a vehicle, determining that a level of confidence of the received information from at least one sensor of a first subset of sensors of the plurality of sensors is less than a first threshold, comparing a number of sensors in the first subset of sensors to a second threshold, and adjusting the driving capability of the vehicle to rely on information received from a second subset of sensors of the plurality of sensors, wherein the second subset of sensors excludes the at least one sensor of the first subset of sensors.