In a focus position control apparatus, a control unit performs, when a size changing amount of an object image is larger than a predetermined amount during control for driving a focus lens according to a distance change of an object image in an optical axis direction based on a determination result of a determination unit, a following driving operation corresponding to the distance change as a driving control, and limits the following driving operation corresponding to the distance change when the size changing amount of the object image smaller than the predetermined amount. The predetermined amount is changed according to an imaging parameter and an object status.