Residual prediction poses a problem that deriving a residual involves a large amount of processing for generating two motion-compensated images and deriving a difference. Provided is a residual prediction device including a reference image interpolation unit that derives two residual prediction motion-compensated images, and a residual synthesis unit that derives a residual from a difference between the two residual prediction motion-compensated images and derives a predicted image by adding the residual to a motion-compensated image, in which the residual synthesis unit derives a predicted image by adding the residual to the motion-compensated image in a case where a target block size is greater than a predetermined size and derives the motion-compensated image as the predicted image in a case where the target block size is less than or equal to a predetermined size.