A system for training a model to predict a sequence (e.g. a sequence of words) given a context is disclosed. A model can be trained to make these predictions using a combination of individual predictions compared to base truth and sequences of predictions based on previous predictions, where the resulting sequence is compared to the base truth sequence. In particular, the model can initially use the individual predictions to train the model. The model can then be further trained over the training data in multiple iterations, where each iteration includes two processes for each training element. In the first process, an initial part of the sequence is predicted, and the model and model parameters are updated after each prediction. In the second process, the entire remaining amount of the sequence is predicted and compared to the corresponding training sequence to adjust model parameters to encourage or discourage each prediction.