Disclosed is an auto focus system having an auto focus (AF) frame auto-tracking function that detects the image of an object to be tracked as an AF target using a pattern matching process and moves an AF frame indicating an AF target range to the position of the object, thereby tracking the object. When a person's face (face image) is included in the AF frame, the auto focus system having an AF frame auto-tracking function automatically sets the AF frame in an appropriate range so as to correspond to the position and size of the face image. Therefore, even when the size of the face image is changed, it is possible to appropriately track the face and perform focusing on the face.