Systems and methods are provided for implementing hybrid prediction. Hybrid prediction integrates two deep learning based trajectory prediction approaches: grid-based approaches and graph-based approaches. Hybrid prediction techniques can achieve enhanced performance by combining the grid and graph approaches in a manner that incorporates appropriate inductive biases for different elements of a high-dimensional space. A hybrid prediction framework processor can generate trajectory predictions relating to movement of agents in a surrounding environment based on a prediction model generating using hybrid prediction. Trajectory predictions output from the hybrid prediction framework processor can be used to control an autonomous vehicle. For example, the autonomous vehicle can perform safety-aware and autonomous operations to avoid oncoming objects, based on the trajectory predictions.