Methods, systems, and apparatus, including computer programs encoded on computer storage media, for controlling a robot to perform a custom real-time action that uses streaming inputs. One of the methods comprises receiving a definition of a custom real-time streaming control function that defines a custom streaming action, wherein the custom streaming action specifies a goal state for a robot in an operating environment; providing a command to initiate the custom streaming action; and repeatedly providing updated goal states for the custom streaming action, wherein the control layer of the framework is configured to execute the custom streaming action including driving the robot toward a most recent goal state at each tick of a real-time robotics control cycle.