Tools and techniques are described to automate triage of security and operational alerts. Insight instances extracted from raw event data associated with an alert are aggregated, vectorized, and assigned confidence scores through classification based on machine learning. Confidence scoring enables heavily loaded administrators and controls to focus attention and resources where they are most likely to protect or improve the functionality of a monitored system. Feature vectors receive a broad base in the underlying instance values through aggregation, even when the number of instance values is unknown prior to receipt of the event data. Visibility into the confidence scoring process may be provided, to allow tuning or inform further training of a classifier model. Performance metrics are defined, and production level performance may be achieved.