A positional shift amount calculation apparatus that calculates a positional shift amount, which is a relative positional shift amount between a first image, based on a luminous flux that has passed through a first imaging optical system, and a second image. A calculation unit calculates a positional shift amount based on data within a predetermined area out of first image data representing first and second image data. A setting unit sets a relative size of the area to the first and second image data. The calculation unit calculates a first positional shift amount using the first and second image data in the area having a first size that is preset. The setting unit sets a second size of the area based on the size of the first positional shift amount and an optical characteristic of the first imaging optical system. The calculation unit then calculates a second positional shift amount.