A system for controlling an irradiating direction of a vehicle lamp includes a target swivel angle calculating section to calculate a target swivel angle based on a vehicle steering angle, a swivel driver to execute, based on the target swivel angle, a swivel control to swivel the irradiating direction, and a fail-safe controller to execute, upon detection of a failure in the system, a fail-safe operation to stop the swivel control. When the system is restored after the failure, the fail-safe controller determines whether the target swivel angle is within a straight-ahead range that includes an angle corresponding to a straight-ahead direction of a vehicle. If the fail-safe controller determines that the target swivel angle is within the straight-ahead range, the fail-safe controller deactivates the fail-safe operation to restart the swivel control.