The technology disclosed relates to building ensemble analytic rules for reusable operators and tuning an operations monitoring system. In particular, it relates to analyzing a metric stream by applying an ensemble analytical rule. After analysis of the metric stream by applying the ensemble analytical rule, quantized results are fed back for expert analysis. Then, one or more type I or type II errors are identified in the quantized results, and one or more of the parameters of the operators are automatically adjusted to correct the identified errors. The metric stream is further analyzed by applying the ensemble analytical rule with the automatically adjusted parameters.