A position controller for an image stabilizing optical element in an optical system includes an advancing/retracting frame which holds part of the optical system, a main actuator, an anti shake movable frame supported by the advancing/retracting frame, an anti shake drive actuator, an insertable/removable frame which holds the image stabilizing optical element and is supported by the anti shake movable frame, a removal drive mechanism which engages with the insertable/removable frame and moves the insertable/removable frame from an insertion position to a removed position, and a controller. Upon an inputter inputting a lens accommodating drive signal to the controller, the controller switches the anti shake drive actuator to a non driving state before moving the insertable/removable frame to the removed position by moving the advancing/retracting frame toward the accommodated position.