An auto-focus device, for eliminating an uncomfortable feeling when phase difference AF is switched to optical path length difference AF when an arbitrary area is set as a focusing target area, and a method for controlling operation of the same. A desired area is set as the focusing target area. Until a first threshold value position is reached, the focus lens is moved based on a phase difference AF evaluation value obtained from a phase difference AF sensor so as to bring the center portion of the imaging area into focus. Until a second threshold value position is reached, the focus lens is moved based on a phase difference AF evaluation value which is obtained from a signal of an area corresponding to the set focusing target area. When the focus lens reaches the second threshold value position, the focus lens is positioned at the focusing position.