An automatic tracking image pickup system including: an image pickup apparatus picking up an image of an object; a driving unit changing an image pickup direction of the image pickup apparatus; a recognition unit recognizing a tracking object in a picked up image; and a controller controlling a speed of the driving unit based on a difference between a position of the tracking object in the image and a target position in the image in an initial mode until the tracking object reaches a predetermined position in the image after the recognition unit recognizes the tracking object for first time, and in a normal mode after the tracking object reaches the predetermined position with a gain for obtaining the speed of the driving unit based on the difference in the normal mode being larger than a gain used in the initial mode.