The image processing apparatus processes an image signal obtained from an image pickup element using an image pickup optical system arranged to obtain directional traveling information of object image light corresponding to a pupil division area of a photographing lens. The apparatus includes: a unit of setting a focal position at which the refocused image is generated; and a unit of correcting an image signal of a defect pixel of the image pickup element using an image signal of other pixel. The apparatus determines the other pixel to be used for correcting the image signal of the defect pixel on the basis of the set focal position and the directional traveling information of the object image light.