Techniques are provided for generating performance improvement recommendations for machine learning models. One method comprises evaluating performance metrics for multiple implementations of a machine learning model; computing a performance score that aggregates the performance metrics for a given machine learning model implementation; and recommending a modification to the given machine learning model implementation based on the performance score by evaluating one or more performance metrics for the given implementation relative to at least one additional performance metric for the given implementation, wherein the recommended modification is based on a performance with the recommended modification for another implementation. A given performance metric may be weighted based on an expected improvement from modifying a factor related to the given performance metric. The recommended modification to the given machine learning model implementation may comprise an indication of the expected improvement for the modification.