Systems and methods for training a neural network for estimating a trajectory of a vehicle are disclosed. In one embodiment, a system includes one or more processors, and a non-transitory computer-readable medium storing computer-readable instructions. The computer-readable instructions cause the one or more processors to receive sensor data of a plurality of examples from a plurality of vehicle sensors, input the sensor data into a sensor data neural network to generate a sensor data intermediate space and receive structured data of the plurality of examples. The computer-readable instructions cause the one or more processors to input the structured data into a structured data neural network to generate a structured data intermediate space, calculate a first loss between the sensor data intermediate space and the structured data intermediate space using a first loss function, and provide the first loss to the sensor data neural network and the structured data neural network.