The amount of time required to train a neural network may be decreased by modifying the neural network to allow for greater parallelization of computations. The computations for cells of the neural network may be modified so that the matrix-vector multiplications of the cell do not depend on a previous cell and thus allowing the matrix-vector computations to be performed outside of the cells. Because the matrix-vector multiplications can be performed outside of the cells, they can be performed in parallel to decrease the computation time required for processing a sequence of training vectors with the neural network. The trained neural network may be applied to a wide variety of applications, such as performing speech recognition, determining a sentiment of text, determining a subject matter of text, answering a question in text, or translating text to another language.