For artifact reduction in a magnetic resonance imaging system, deep learning trains an image-to-image neural network to generate an image with reduced artifact from input, artifacted MR data. For application, the image-to-image network may be applied in real time with a lower computational burden than typical post-processing methods. To handle a range of different imaging situations, the image-to-image network may (a) use an auxiliary map as an input with the MR data from the patient, (b) use sequence metadata as a controller of the encoder of the image-to-image network, and/or (c) be trained to generate contrast invariant features in the encoder using a discriminator that receives encoder features.