Methods, systems, and apparatus, including computer programs encoded on computer storage media, for constructing and processing computational graphs that represent dynamically structured machine learning models are disclosed. An example system receives data identifying a plurality of operations that can be performed on input data for processing by a dynamically structured machine learning model. The system also receives a plurality of labels corresponding to arguments for the plurality of operations. A directed computational graph representing a comprehensive layer of the dynamically structured machine learning model is generated from the identified operations and labels. An example system then receives an input for processing by the machine learning model and specifies data flow through the directed computational graph.