In an image processing apparatus, three-dimensional shape data indicating a shape of an object is generated based on at least one of a plurality of images captured from different directions by a plurality image capturing devices. Color information indicating a color of each element of the three-dimensional shape data is generated based on the generated three-dimensional shape data and the plurality of images. The three-dimensional shape data and the color information are associated with each other and written to a storage unit as material data for generating a virtual viewpoint image corresponding to virtual viewpoint information indicating a virtual viewpoint.