An autofocus system which can prevent erroneous tracking of an unintended tracking target, is provided. In a recognition and tracking mode which uses an AF frame to automatically track a face of a specific person for which certification data is previously stored, a face of a specific person designated by an operator is detected by face authentication processing from a captured image obtained from a television camera. Thereafter, the face is detected from the captured image by a pattern matching processing in which an image of the face is used as a reference pattern for object tracking processing, and the AF frame is moved to the detected position. The face recognition processing is executed each time when the object tracking processing is executed a predetermined times, thereby preventing erroneous tracking by the object tracking processing.