Image data obtained by imaging of an imaging element capable of imaging a subject with sensitivity to a wavelength band of visible light and a wavelength band of near-infrared light via an optical system is acquired. A point image restoration process using a common restoration filter is performed on the image data of the subject captured with sensitivity to the wavelength band of the visible light by the imaging element and the image data of the subject captured with sensitivity to the wavelength band of the near-infrared light by the imaging element. The common restoration filter is calculated on the basis of average optical characteristics of the optical system obtained by performing weighted averaging of first optical characteristics with respect to the visible light of the optical system and second optical characteristics with respect to the near-infrared light of the optical system.