An image processing apparatus comprising: a foreign substance information acquisition unit which acquires, from input moving image data, foreign substance information containing a position and size of a foreign substance adhering to an optical element arranged in front of an image sensor in an image capturing apparatus that captures the moving image data; a motion vector information acquisition unit which acquires, from the moving image data, a motion vector of a macroblock obtained by dividing the moving image data; a correction playback unit which corrects deterioration of image quality caused by the foreign substance, based on the foreign substance information, and plays back moving image data; and a determination unit which determines, based on a motion vector of a macroblock matching the position of the foreign substance in the moving image data to be played back, whether to perform foreign substance correction playback of the macroblock.