For heart segmentation in magnetic resonance or other medical imaging, deep learning trains a neural network. The neural network, such as U-net, includes at least one long-short-term memory (LSTM), such as a convolutional LSTM. The LSTM incorporates the temporal characteristics with the spatial to improve accuracy of the segmentation by the machine-learnt network.