An arithmetically operating device is provided and includes: an image acquiring portion configured to acquire phase difference images of a sample; a retrieval range setting portion configured to set a range of pixels each to be made a retrieval object in the other image in the phase difference images for pixels each to be made a representative set in one image in the phase difference images, and pixels other than the pixels each to be made the representative; a correlation calculating portion configured to calculate correlations between the pixels for the one image, and the pixels in the range set for the pixels; and a parallax calculating portion configured to calculate parallaxes for the pixels for the one image in accordance with the correlations thus calculated.