Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing an input sequence using a recurrent neural network to generate an output for the input sequence. One of the methods includes receiving the input sequence; generating a doubled sequence comprising a first instance of the input sequence followed by a second instance of the input sequence; and processing the doubled sequence using the recurrent neural network to generate the output for the input sequence.