A performance optimizing circuit is provided for a signal processing system which is parameterized by a set of coefficients that vary the operational characteristics of the signal processing system. The performance optimizing circuit receives as input a reference signal and an output signal of the signal processing system. The performance optimizing circuit may include (a) a cost computation circuit that receives the reference signal and the output signal and provides as output a cost signal representing a cost function computed using a set of current values for the set of coefficients, the output signal and the reference signal; and (b) a cost optimizer circuit that, at each of a plurality of successive time intervals, evaluates one or more values of the cost signal in the cost computation circuit and provides to the signal processing system a new set of values for the set of coefficients. The cost optimizer circuit implements two or more of the random search, parabolic interpolation and hill climbing techniques.