The present application discloses a method to optimize operation of a transformer cooling system, the corresponding cooling system, and a method to determine the capacity of Variable Frequency Drives (VFD) that are used in the transformer cooling system. The method comprises: preprocessing the initial data input by user; collecting the on-line data, and calculating the optimized control command to meet the requirement of the transformer loss, top-oil temperature variation and noise; and executing the control actions by controlling a controllable switch and/or sending a control command to a VFD. Compared with the existing prior arts, the proposed solutions are much more intuitive and practical in the field of the cooling system.