Methods, computer systems, and computer storage media are provided for evaluating information retrieval (IR) such as search query results (including advertisements) by a machine learning scorer. In an embodiment, a set of features is derived from a query and a machine learning algorithm is applied to construct a linear model of (query, ads) for scoring by maximizing a relevance metric. In an embodiment, the machine learned scorer is adapted for use with WAND algorithm based ad selection.