Described is a system for continual learning using experience replay. In operation, the system receives a plurality of tasks sequentially, from which a current task is fed to an encoder. The current task has data points associated with the current task. The encoder then maps the data points into an embedding space, which reflects the data points as discriminative features. A decoder then generates pseudo-data points from the discriminative features, which are provided back to the encoder. The discriminative features are updated in the embedding space based on the pseudo-data points. The encoder then learns (updates) a classification of a new task by matching the new task with the discriminative features in the embedding space.