Methods, systems, apparatus, including computer programs encoded on computer storage medium, for generating a sentence summary. In one aspect, the method includes actions of tokenizing the sentence into a plurality of tokens, processing data representative of each token in a first order using an LSTM neural network to initialize an internal state of a second LSTM neural network, processing data representative of each token in a second order using the second LSTM neural network, comprising, for each token in the sentence: processing the data representative of the token using the second LSTM neural network in accordance with a current internal state of the second LSTM neural network to (i) generate an LSTM output for the token, and (ii) to update the current internal state of the second LSTM neural network, and generating the summarized version of the sentence using the outputs of the second LSTM neural network for the tokens.