Described are techniques for modifying or creating classification data used to automatically classify items in an online marketplace or catalog, based on user interaction data. For one or more classification labels that may be applied to an item, user interaction data indicative of a count of instances that the label was accessed, a length of time during which the label was accessed, counts of instances that parent and child labels were accessed, and counts of instances that the label was accessed via a search query may be determined. Based on the user interaction data, an importance score for the label may be determined. Labels having an importance score greater than or equal to a threshold value may be included in classification data and used for subsequent classification of items. Labels having an importance score less than a threshold may be excluded from the classification data.