An image processing apparatus includes an acquisition unit configured to acquire color information about an object and color information about a background in a first image captured by an image capturing apparatus, a storage unit configured to store the acquired color information about the object and the acquired color information about the background, an expansion unit configured to expand, in a three-dimensional color space, the color information about the object and the color information about the background stored in the storage unit, and an extraction unit configured to extract an area of the object from a second image captured by the image capturing apparatus, based on the respective pieces of the color information expanded by the expansion unit.