A method comprises determining a vectorized representation of positions of road agents and road geometry based on sensor data from a vehicle, inputting the vectorized representation of the positions of the road agents and the road geometry into a trained transformer network, predicting one or more road agent trajectories at one or more future time steps based on an output of the transformer network, predicting an acceleration of the vehicle at the one or more future time steps based on the predicted one or more road agent trajectories at the one or more future time steps, and causing the vehicle to perform the predicted acceleration at the one or more future time steps.