A three-dimensional imaging device (10) according to an aspect of the present invention, which outputs first and second viewpoint images due to a pupil division on a light flux from a single imaging optical system (12, 14), includes a defocus map calculation unit (61) calculating a defocus map representing a defocus amount at each position in the first and second viewpoint images; a restoration filter storage (62) storing restoration filters corresponding to the defocus amount and an image height at each position in the viewpoint images, the restoration filters being deconvolution filters based on a point image intensity distribution representing an imaging performance of the imaging optical system; and a restoration unit (63) selecting a restoration filter, which corresponds to the defocus amount and the image height at each position in the viewpoint images, for each position in the first and second viewpoint images, and restoring the first and second viewpoint images by performing a deconvolution for each position in the first and second viewpoint images based on the selected restoration filter.