In some examples, a first set of image data is received, the first set of image data corresponding to images of a first type and being of a person in an environment of a vehicle and including a first plurality of images of the person over a time interval. In some examples, a second set of image data is received, the second set of image data corresponding to images of a second type and being of the person in the environment of the vehicle and including a second plurality of images of the person over the time interval. In some examples, the first set of image data and the second set of image data are processed to determine a recognized action of the person, which includes using a first neural network to determine the recognized action of the person.