A controller for controlling a vapor compression system is provided. The controller is configured to control an operation of the VCS with different combinations of setpoints for different actuators of the VCS to estimate a cost of operation of the VCS for each of the different combinations of setpoints, and compute, using a Bayesian optimization of the combinations of setpoints and their corresponding estimated costs of operation, a probabilistic surrogate model, wherein the probabilistic surrogate model defines at least first two order moments of the cost of operation in the probabilistic mapping. The controller is further configured to select an optimal combination of setpoints having the largest likelihood of being a global minimum at the surrogate model according to an acquisition function of the first two order moments of the cost of operation.