System and method for explaining driving behavior actions of autonomous vehicles. Combined sensor information collected at a scene understanding module is used to produce a state representation. The state representation includes predetermined types of image representations that, along with a state prediction, are used by a decision making module for determining one or more weighted behavior policies. A driving behavior action is selected and performed based on the determined one or more behavior policies. Information is then provided indicating why the selected driving behavior action was chosen in a particular driving context of the autonomous vehicle. In one or more embodiments, a user interface is configured to depict the predetermined types of image representations corresponding with the driving behavior action performed via the autonomous vehicle.