An online system maintains machine learning models that determine risk scores for content items indicating likelihoods of content items violating content policies associated with the machine learning models. When the online system obtains an additional content policy, the online system applies a maintained machine learning model to a set including content items previously identified as violating or not violating the additional content policy. The online system maps the risk scores determined for content items of the set to likelihoods of violating the additional content policy based on the identifications of content times in the set violating or not violating the additional content policy. Subsequently, the online system applies the maintained machine learning model to content items and determines likelihoods of the content items violating the additional content policy based on the mapping of risk scores to likelihood of violating the additional content policy.