In one embodiment, a system for determining the direction of objects in an image is provided. An image is received from a camera associated with a vehicle. The image is processed to identify an object, such as another vehicle. To determine the direction the object is traveling, the image is provided as input to a machine learning algorithm that outputs a direction of the detected object based at least in part on a training dataset including a plurality of images annotated to include respective direction information determined based at least in part on a plurality of 3D models set to aligned orientations that are significantly aligned with orientations of objects captured in the plurality of images. The objects and their associated directions can be provided to one or more autonomous vehicle functions, such as hazard detection or navigation.