According to one embodiment, a method, computer system, and computer program product for retraining a classifier-based automatic dialog system with recorded user interactions is provided. The present invention may include receiving recorded interactions, where the interactions are between a user and an automatic dialog system; determining, based on the recorded interactions, whether to pair a given input with one or more classes; pairing inputs with one or more classes; assessing the reliability of the paired inputs and classes; determining whether the reliable paired inputs and classes can be consistently mapped; and merging all consistently mapped reliable pairs with an initial training set.