The present disclosure relates techniques for autonomously controlling heavy equipment and vehicles using task hierarchies. Particularly, aspects of the present disclosure are directed to obtaining a task to be performed by an autonomous vehicle, determining subtasks to be performed to perform the task, obtaining sensor data providing a representation of operation of the autonomous vehicle in a worksite environment and situational context of the worksite environment, determining a task context for a subtask based on the sensor data, identifying a predictive model from a library of predictive models based on the task context, estimating, by the predictive model, a set of output data based on sensor data, and controlling operations of the autonomous vehicle in the worksite environment to perform the subtask using a set of input data derived from the sensor data and the set of output data.