Systems and methods for controlling a vehicle based on driver engagement are disclosed. In one embodiment, a method of controlling a vehicle includes determining, using driver data from one or more driver sensors, a driver state estimation of a driver of the vehicle, determining, using environment data from one or more environmental sensors, one or more environment anomalies within an environment of the vehicle, and determining an anomaly category for at least one of the one or more environment anomalies. The method further includes, based on the driver state estimation and one or more anomaly categories, selecting a failure mode, based on the failure mode, selecting at least one failsafe action, and determining an operation of the vehicle in accordance with the failure mode.