The present disclosure provides a system for improving performance of a neural network model. The system receives the neural network model and a training data associated with the neural network model. In addition, the system examines a first plurality of neuron activations inside the neural network model for the training data. The system examines the first plurality of neurons for creating a statistical profile of the first plurality of neuron activations. Further, the system receives a new set of data samples to improve the neural network model. Furthermore, the system examines a second plurality of neuron activations of each new sample of the new set of data samples. Moreover, the system extracts one or more data samples from the new set of data samples with largest novelty measurements. Also, the system adds the extracted one or more samples to the training data for re-training of the neural network model.