In one embodiment, a device in a network detects an anomaly in the network by analyzing a set of sample data regarding one or more conditions of the network using a behavioral analytics model. The device receives feedback regarding the detected anomaly. The device determines that the anomaly was a true positive based on the received feedback. The device excludes the set of sample data from a training set for the behavioral analytics model, in response to determining that the anomaly was a true positive.