The disclosed embodiments provide a system for streamlining machine learning. During operation, the system determines a resource overhead for a baseline version of a machine learning model that uses a set of features to produce entity rankings and a number of features to be removed to lower the resource overhead to a target resource overhead. Next, the system calculates importance scores for the features, wherein each importance score represents an impact of a corresponding feature on the entity rankings. The system then identifies a first subset of the features to be removed as the number of features with lowest importance scores and trains a simplified version of the machine learning model using a second subset of the features that excludes the first subset of the features. Finally, the system executes the simplified version to produce new entity rankings.