There is provided a focus detection device including a phase difference acquisition unit that calculates an amount of deviation between light-reception amount distributions of a pair of light-receiving element groups arranged in a predetermined direction perpendicular to an optical axis direction as a phase difference, a conversion coefficient correction unit that corrects a conversion coefficient, representing a ratio of an amount of focus deviation in the optical axis direction to the phase difference when shapes of the light-reception amount distributions are same, according to a degree of discrepancy between the shapes of the light-reception amount distributions, and a defocus amount generation unit that generates the amount of focus deviation as an amount of defocus based on the corrected conversion coefficient and the phase difference.