An image stabilizing apparatus includes one or more processors, and a memory storing instructions which, when executed by the one or more processors, cause the image stabilizing apparatus to function as a detecting unit configured to detect a motion vector in a plurality of areas made by dividing motion vector detecting frames set in a plurality of images sequentially output from an imaging unit, and a control unit configured to determine a motion vector of an object in the detected motion vector and to perform an image stabilization control based on the motion vector of the object. The control unit determines a size of the object based on the motion vector of the object, and changes the motion vector detecting frame based on the size of the object and a position of the object.