An image processing apparatus acquires information about a three-dimensional display apparatus, acquires a plurality of images based on imaging by a plurality of imaging apparatuses that images a target area from different directions, sets a position of and a direction from a virtual viewpoint in a space associated with the target area, sets positions of and directions from virtual viewpoints of a number corresponding to a configuration of the three-dimensional display apparatus, with the position of and the direction from the virtual viewpoint as a reference, and generates display image data for the three-dimensional display apparatus, based on the plurality of images and the set positions of and the set directions from the virtual viewpoints.