An image processing apparatus acquires an input image generated by image pickup via an optical system, and perform unsharp mask processing for the input image using a filter generated based on information of a PSF of the optical system. The filter is generated based on an unsharp mask used for the unsharp mask processing, and includes two-dimensional data having filter coefficients that are arranged rotationally asymmetrically with respect to a filter coefficient corresponding to a target pixel in the input image in convoluting the filter with the input image. A peak position or a center of gravity position of the unsharp mask accords with a position of the filter coefficient corresponding to the target pixel in the filter.