Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an industrial plant controller that controls operation of an industrial plant. In one aspect, a method comprises generating training data using an industrial plant simulation model that simulates operation of the industrial plant. The industrial plant controller is trained by a reinforcement learning technique using the training data. The industrial plant controller is configured to process an input comprising a state vector characterizing a state of the industrial plant in accordance with a plurality of industrial plant controller parameters to generate an action selection policy output that defines a control action to be performed to control the operation of the industrial plant.