Disclosed is an image processing apparatus including: a sharpness recovery unit configured to recover sharpness of an input image; a correlation determination unit configured to determine a correlation between a target pixel and peripheral pixels of the input image; a noise removal unit configured to perform smoothing of the target pixel determined as having a correlation with the peripheral pixels in a recovered image output from the sharpness recovery unit; and a switching unit configured to switch between a mode in which the correlation determination unit uses the target pixel and the peripheral pixels of the pre-recovery input image in the correlation determination and a mode in which the correlation determination unit uses the target pixel and the peripheral pixels of the recovered input image in the correlation determination, depending on recovery characteristics used in the determination of the correlation determination unit.