An image processing apparatus comprises a unit configured to set a color selected by a user from among colors included in scanned image data obtained by reading a reference sample as a destination color for the adjustment-target color. Then, the unit configured to set a destination color has a function to set the destination color based on first scanned image data obtained by reading a color sample as the reference sample, in which patches of a plurality of colors including colors around the adjustment-target color are printed and a function to set the destination color based on second scanned image data obtained by reading a sample document.