An imaging device, includes: an imaging element including imaging pixels and phase difference detecting pixels, a calculating unit which calculates numerical information corresponding to a defocus amount using an image signal obtained by imaging by the phase difference detecting pixels whenever the imaging is performed by the imaging element, a difference calculating unit calculating with respect to at least three times of imaging which is continuously performed, a difference between the numerical information obtained by arbitrary imaging and the numerical information obtained by the imaging which is continuously performed after the arbitrary imaging, and a control unit which moves a focus lens to a focus position using the obtained numerical information when a variance of the obtained differences is smaller than a threshold value and controls a position of the focus lens not to be changed when the variance is equal to or larger than the threshold value.