There is provided an image processing device including a depth acquisition unit and a smoothing processing unit. The depth acquisition unit acquires a depth to a subject in correlation with a pixel in a captured image of the subject, and the smoothing processing unit designates a pixel in a region excluding a predetermined region in the image as a target pixel, and performs a smoothing process of the degree according to the depth corresponding to the target pixel for a pixel value of the target pixel in a predetermined direction. This causes a pixel in a region excluding the predetermined region in the image to be blurred, thereby generating a panning image.