Technologies are described herein that pertain to optimizing a ranker component for a risk-oriented objective. Various definitions of risk are described herein, wherein risk is based upon variance in performance scores assigned to the ranker component for respective queries in a data store. Additionally, risk is optionally based upon variance in relative performance of the ranker component versus a baseline ranker component.