To provide a scheme that, when making a prediction in multilayer coding, takes into account the quality of the image in each layer and can adaptively determine in want manner to refer to the image of which layer. Provided is an image processing device provided with a prediction control unit that, on the basis of an indicator dependent on the image quality of a first layer image and a second layer image, sets the weighting when including a first layer pixel and a second layer pixel for predicting a pixel in the second layer that is encoded or decoded referring to the first layer.