A content hosting system determines whether or not to remove media content items that use, either in whole or in part, media content of other media content items. The media content items are uploaded or otherwise added to the system by different owners. To correctly make this determination appropriately, the content hosting system adopts a two-step process. First, for a given probe media content item, the content hosting system identifies reference media content items of which reference media content is reused in the probe media content item. Second, the content hosting system evaluates whether or not the probe media content item should be removed. To make this evaluation, the content hosting system uses contextual information such as characteristics associated with a reference media content item and the probe media content item.