An image processing apparatus that corrects chromatic aberration that occurs in a display optical system, using a display unit in which respective pixels have a plurality of primary color elements that emit light of respectively differing primary colors, the apparatus comprises an acquisition unit adapted to acquire a displacement amount in which the position of a primary color element used as a light source is used as a reference, for light of each primary color emitted by a primary color element of the display unit, a calculation unit adapted to calculate a display position in the display unit for each primary color component of each pixel in a display image, based on the acquired displacement amount, and a display control unit adapted to display each primary color component of each pixel of the display image at the display position of the display unit calculated by the calculation unit.