A disparity calculating apparatus which calculates a disparity value from a stereo image including first and second images includes: an image dividing unit which divides the first image into segments; a reference point determining unit which sets reference points to the first image; a corresponding point detecting unit which (i) calculates, for each reference point, a corresponding point which is included in the second image and corresponds to the reference point, based on phase information of images obtained by performing Fourier transform on the first image and the second image, (ii) calculates a disparity value using the reference point and each corresponding point, and (iii) calculates a reliability of the disparity value; and a disparity assigning unit which assigns each of the segments which includes a corresponding one of the reference points the disparity value and the reliability calculated by the corresponding point detecting unit.