A control system controls a machine according to a control policy parameterized on a lifted state space of the machine having an unknown onto mapping to a state space of the machine. The control system accepts measurements of state variables that form a portion of the state of the machine and determines a derivative of at least one measured state variable using values of the state variable measured for multiple time instances, such that a combination of the measured state variables and the derivative of the at least one measured state variable defines the lifted state for the time instance. The control system updates the control policy by evaluating a value function of the control policy using the lifted states, such that a control input to the machine is determined using the lifted state and the updated control policy.