An image pickup apparatus on which a lens unit is to be removably mounted includes an image pickup element, a first focus detection unit which performs a focus detection by a contrast detection method based on a signal output from the image pickup element, a second focus detection unit which performs a focus detection by a phase difference detection method based on a pair of image signals output from the image pickup element, and a control unit which performs a focus control based on an in-focus position detected by the first or second focus detection unit, and the control unit receives, from the mounted lens unit, first information related to a displacement of the in-focus position by the phase difference detection method and determines whether to use the second focus detection unit for the focus control according to the first information.