An online system determines a stability metric that indicates overlap between the set of entities associated with a particular entity when embeddings have been adjusted due to modifications in the input data of an embedding model. The online system generates a stability score for the embedding model by computing a statistic for one or more stability metrics. The online system determines a stability metric for a particular content provider by identifying a first cluster of content providers in a set of first embeddings, and a second cluster of content providers in a set of second embeddings. The second embeddings are generated after modifications have been made to input data. The online system determines the stability metric based on an overlap between the first cluster and the second cluster of content providers. The stability score can be an indicator of model performance that can be used to select embedding models.