This application is directed to aerial view generation for vehicle control. A vehicle obtains a forward-facing view of a road captured by a front-facing camera of a vehicle and applies a machine learning model to process the forward-facing view to predict determine a trajectory of the vehicle and a road layout based on a Frenet-Serret coordinate system of the road for the vehicle. The trajectory of the vehicle is combined with the road layout to predict an aerial view of the road, and the aerial view of the road is used to at least partially autonomously drive the vehicle. In some embodiments, the machine learning model is applied to process the forward-facing view to determine a first location of each of an obstacle vehicle in the Frenet-Serret coordinate system. The first location of the obstacle vehicle is converted to a vehicle location on the aerial view of the road.