There is provided a control unit of an optical apparatus and an image pickup apparatus, which can prevent a shift of an already controlled position of a member when an impact, etc. apply to the optical apparatus. A control unit for an optical apparatus that includes a first member that is movable, an actuator, and a drive mechanism that transmits a driving force of the actuator to the first member, includes a determination part for determining whether the optical apparatus or a second member provided to an apparatus attached to the optical apparatus is to move, and a controller for controlling, prior to a movement of the second member, the actuator from a first state at which the drive mechanism can transmit the driving force to a second state at which the drive mechanism cannot transmit the driving force, when the determination part determines that the second member is to move.