Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for intervention behavior prediction. One of the methods includes receiving data characterizing a scene that includes a first agent and a second agent in an environment. A confounder prediction input generated from the data is processed using a confounder prediction model. A plurality of predicted conditional probability distributions is generated, wherein each predicted conditional probability distribution is conditioned on: (i) a planned intervention by the second agent, and (ii) the confounder variable belonging to a corresponding confounder class. An intervention behavior prediction for the first agent is generated based on the plurality of the predicted conditional probability distributions and the confounder distribution, wherein the intervention behavior prediction includes a probability distribution over the plurality of the possible behaviors for the first agent in reaction to the second agent performing the planned intervention.