An image photographing apparatus is provided. The image photographing apparatus includes a first photographing unit configured to photograph an image by using a phase difference image sensor, a second photographing unit configured to be spaced on a side of the first photographing unit, a controller configured to generate a first depth map by using a first image photographed by the first photographing unit and generate a second depth map by using a second image photographed by the second photographing unit, and an image processor configured to generate a three-dimensional (3D) image by using the first depth map and the second depth map.