A focus adjustment device includes a direction judgment unit and a control unit. A direction judgment unit calculates an evaluation value based on an image signal of a focus detection region, thereby judging a drive direction of a focus lens. The control unit controls a focus adjustment operation on the basis of the drive direction. The control unit causes the direction judgment unit to repeatedly judge the drive direction, and when the focus lens is slightly driven in a first direction judged on the basis of a first evaluation value and then the focus lens is slightly driven in a second direction judged on the basis of a subsequently calculated second evaluation value, the control unit forbids the slight driving of the focus lens in the first direction even though a drive direction judged on the basis of a further subsequently calculated third evaluation value is the first direction.