According to the apparatus and method of the present invention, the motion vector Vij is weighted by the weighting coefficient αij of the motion vector Vij decided based on the AF evaluation value Iij indicating the degree of sharpness of the subject image so as to decide the entire motion vector V. For this reason, it is possible to reduce an influence on the decision of the motion vector V of the entire image exerted by the motion vector of the object of a low degree of sharpness, that is, the object not focused on.