Both of a high speed AF using contrast control and a high precision AF using a color temperature sensor are achieved. A focus detection device has: a first sensor which outputs a signal for a focus detection by detecting a phase difference; and a second sensor having a plurality of pixels whose spectral sensitivities are different. Accumulation start timing of the first sensor and that of the second sensor are different. The first sensor controls an accumulation time period and a read-out gain based on a difference between the maximum and minimum values of an accumulation signal. The accumulation of the second sensor is finished in a timing when the accumulation of the first sensor is finished.