An image processing apparatus calculates correlation values between a plurality of images with respect to relative positions, and detects a motion vector between the plurality of images based on the calculated correlation values. The image processing apparatus then determines the reliability of the motion vector based on a value indicating the highest correlation among the correlation values and a correlation value obtained at a position that is distant by a predetermined distance or longer from a position at which the value indicating the highest correlation is obtained. In this way, the reliability of the motion vector can be determined with high accuracy.