There is provided with an image processing apparatus. A determination unit determines color information of an object present in an object image from a virtual viewpoint. The color information of the object is determined by combining color information of the object in a captured image obtained by one or more image capturing apparatus selected based on position and orientation information of the virtual viewpoint, position and orientation information of the image capturing apparatuses, and position information of the object, by using a weight corresponding to an image capturing direction of the one or more image capturing apparatus and a position of the object in a field of view of the one or more image capturing apparatus.