Techniques for selecting models using greedy search on validation metrics are disclosed herein. A system generates corresponding predictions for a validation dataset using a plurality of prediction models. The system selects one of the prediction models for inclusion in an ensemble set based on the selected prediction model generating more correct predictions for the validation dataset than the other prediction models, and then removes the selected prediction model from the plurality of prediction models to form a reduced plurality of prediction models. The system identifies remaining data records of the validation dataset for which the selected prediction model generated an incorrect prediction, and then selects an additional prediction model from the reduced plurality of prediction models for inclusion in the ensemble set based on a determination that the additional prediction model generated more correct predictions for the remaining data records than the other prediction models in the reduced plurality of prediction models.