There is provided an image processing apparatus including: a prediction tap selection unit which selects a pixel which is a prediction tap used for prediction operation for acquiring a pixel value of a target pixel which is a target from a second image obtained by converting a first image, from the first image; a classification unit which classifies the target pixel to any class from a plurality of classes; a tap coefficient output unit which outputs a tap coefficient of a class of the target pixel from tap coefficients, acquired by learning to minimize an error between a result of the prediction operation using a student image corresponding to the first image and a teacher image corresponding to the second image; and an operation unit which acquires a pixel value of the target pixel by performing the prediction operation using the tap coefficient and the prediction tap.