An inertial drive actuator includes a shift unit that generates a small shift, a vibration base plate that moves to and fro with the small shift of the shift unit, a mover having a first magnetic field generating unit, a first yoke, and a second yoke. The first yoke and an end portion of the second yoke are opposed to each other at a position outside the vibration base plate, thereby restricting movement of the mover with respect to the direction perpendicular to the driving direction. The frictional force acting between the mover and the vibration base plate is controlled by controlling a magnetic field generated by the first magnetic field generating unit to drive the mover.