While a focus lens is moving to a focus lens driving target position, a controller of a focus adjustment device switches a moving speed of the focus lens from a first speed to a second speed. In particular, the controller calculates the number of times the defocus amount can be calculated when the focus lens moves at a predetermined speed in a section between a first position in which the moving speed is switched and a second position in which the focus lens starts being controlled to stop in the focus lens driving target position, and calculates the second speed based on the number of times, the first position and the focus lens driving target position.