A light source estimating device includes: a distribution holding section holding a spectral energy distribution of a specific light source as a specular reflection component; a set generating section generating a set determined by the specular reflection component and pixel data of the input image in a predetermined space, with respect to each pixel data of the input image; a detecting section detecting, with respect to each pixel data of the input image, another pixel data neighboring the pixel data and included in the set; and a determining section determining whether or not the specific light source corresponds to the light source used during capture, by setting a likelihood that the specific light source corresponds to the light source used during capture high if the other pixel data exists, and setting a likelihood that the specific light source corresponds to the light source used during capture low if the other pixel data does not exist.