Systems and methods for training a residual neural network are described. One of the methods includes: initializing one or more weights of each of a plurality of residual units; inputting a plurality of training cases to a first warp unit in a series; using each training case to optimize the one or more weights for each residual unit in parallel in the first warp unit in the series; starting with the output of the first warp unit in the series, iteratively propagating the output of each warp unit to the input of a next respective warp unit in the series, for each respective warp unit, using each training case to optimize the one or more weights for each residual unit in parallel in the respective warp unit; and storing the output of the last warp unit in the series and the weights for each residual unit.