A method for training an image colorization model may include inputting a training input image into a colorization model and receive a predicted color map as an output of the colorization model. A first color distance may be calculated between a first pixel of the predicted color map and a second pixel of the predicted color map. A second color distance may be calculated between a third pixel included in a ground truth color map and a fourth pixel included in the ground truth colorization map. The third pixel and fourth pixel included in the ground truth color map may spatially correspond, respectively, with the first pixel and second pixel included in the predicted color map. The method may include adjusting parameters associated with the colorization model based on a neighborhood color loss function that evaluates a difference between the first color distance and the second color distance.