A focus adjustment device includes a focus detection region setting section, a direction determination section and control section. The focus detection region setting section sets focus detection regions which are different in size from one another. The direction determination section determines a movement direction of a focus lens to be in focus based on a change of the contrast caused by the movement of the focus lens. The control section repetitively determines whether determination results of the movement direction obtained in the respective focus detection regions are different. The control section inhibits a switchover of a focus adjustment operation when it is consecutively determined a predetermined number of times or more that the determination results of the movement direction are different, otherwise the control section does not inhibit the switchover.