An image processing device includes: a smoothing section configured to extract a smoothing tap and smooth a target image on the basis of pixel values within the tap, the smoothing tap being of variable size and including plural pixels centered on each target pixel of the image; a class tap extracting section configured to extract a class tap including plural pixels centered on each target pixel in the smoothed image; a class code determining section configured to generate a code corresponding to a characteristic of variation of pixel values within the class tap, and determine a class code including a size of the smoothing tap and the code; and a pixel value computing section configured to read tap coefficients corresponding to the determined class code, and multiply pixel values forming a prediction tap extracted from the smoothed image, by the tap coefficients to calculate pixel values of a processed image.