Systems and methods for analyzing a driver's brake behavior are provided. In some aspects, the system includes a memory that stores instructions for executing processes for analyzing the driver's brake behavior, and a processor configured to execute the instructions. In aspects, the processes include: receiving data from one or more multi-modal sensors, determining whether each braking event of a plurality of braking events is a hard braking event; analyzing the one or more physiological signals of the driver to determine whether each of the hard braking events is a reactive hard braking event or an intended hard braking event; applying a Lasso regression analysis and a classification framework for each of the hard braking events to identify features that indicate whether a respective hard braking event is a reactive hard braking event or an intended hard braking event; and generating a signal notifying a vehicle of the features.