A solid-state imaging device has normal pixels, first phase difference pixels, and second phase difference pixels. “R”, “G”, and “B” denote colors of color filters provided for the respective pixels. A color dependence characteristic calculator averages output signals from the normal pixels located at the center of a screen, on a color basis, and thereby calculates average output signals. A color determination section determines a main color of a subject, out of three primary colors, based on the average output signals. An AF controller performs focus adjustment of a taking lens based on the output signals of the determined main color, out of the output signals from the first phase difference pixels and the second phase difference pixels.