An image restoration device according to the present invention includes: a point spread function storage unit which stores any two PSFs among a first range PSF for restoring a first sweep image serving as a sweep image when a sweep range is a first range; a second range PSF for restoring a second sweep image serving as the sweep image when the sweep range is a second range which excludes the first range; a third range PSF for restoring a third sweep image serving as the sweep image when the sweep range is a third range obtained by combining the first range and the second range; and a PSF calculation unit which performs addition or subtraction on the two PSFs to calculate a remaining one among the first, second, and third range PSFs.