An image capturing apparatus includes a driving unit for displacing one of a lens unit and an imaging device with respect to an optical axis, and a controller for causing the driving unit to displace one of the lens unit and the imaging device, in response to the motion of the image capturing apparatus. When, with a shooting direction being moved, captured images are generated so that a panoramic image is generated from the captured images, the controller displaces the position of the lens unit or the imaging device at the start of exposure of each image to be captured in a direction based on a direction in which the shooting direction is moved. In addition, a displacement amount at the start of the exposure is set in response to the length of an exposure time period so that the displacement during the exposure time period becomes small and the quality of an image is improved.