An image forming apparatus includes an image sensor comprising a plurality of pixels at least part of which form a plurality of focus detection pixel pairs, each receiving light rays passing through different pupil areas of an image forming optical system; a detection unit configured to detect a defocus amount, based on a phase difference between signals output from the focus detection pixel pairs in a focusing area; a judging unit configured to determine if a subject corresponds to a predetermined pattern, based on the signals output from the pixels other than the focus detection pixel pairs around the focus detection pixel pairs in the focusing area; and a focus control unit configured to drive the image forming optical system to be in an in-focus state based on the detected defocus amount. The judging unit changes focus control, performed by the focus control unit, based on the determination result.