A computing device receives a training data set that includes a plurality of positive examples of sensitive data and a plurality of negative examples of sensitive data. The computing device analyzes the training data set using machine learning to generate a machine learning-based detection (MLD) profile that can be used to classify new data as sensitive data or as non-sensitive data. The computing device computes a quality metric for the MLD profile.