An automatic ball tracking camera for taking video images of a ball game includes a two-stage video camera unit including two sets of camera modules arranged on top of each other, each camera module including a video camera, a zoom lens, and a pan-tilt pod. A video switcher selects one video camera as output and an image processor analyzes an video image currently in use for output and determines whether a ball position moves out from a preset area defined on that video image. A controller determines a target viewing direction to recapture the ball within the preset area in a close-up mode, adjusts the video camera currently not in use for output to achieve the target viewing direction, and switches the output to the video camera adjusted when determined that the ball position moves out from the preset area.