A focus adjustment device, that provides a plurality of detection areas in an imaging region, repeatedly generates ranging values for a physical object in the detection areas, and performs focus adjustment for the physical object based on the ranging values, including a processor that has a statistical processing section, a movement state estimating section, a prediction section, and a control section, wherein the statistical processing section subjects ranging values of the plurality of detection areas to statistical processing, the movement state estimating section calculates representative values that contain statistical dispersion based on the statistical processing, and estimates movement state of the physical object based on time-series change in the representative value, the prediction section predicts focal position regarding the physical object based on the movement state that has been estimated, and the control section performs focus adjustment based on the focal position that has been predicted.