An image processing device includes a phase difference detection portion configured to detect phase difference between parallax images by performing correlation value calculation with a plurality of parallax images, and generate phase difference distribution in an image. The phase difference detection portion performs the phase difference detection individually along two or more directions different from each other, and generates the phase difference distribution by utilizing results of the phase difference detection regarding the two or more directions.