An autonomous driving device includes an autonomous driving release unit configured to release the autonomous driving control if an override operation is input during the execution of the autonomous driving control which is based on a first travel plan, a travel plan setting unit configured to set a second travel plan different from the first travel plan after the input of the override operation, an autonomous driving restore determination unit configured to determine whether or not to restore the autonomous driving control with the autonomous driving control which is based on the second travel plan after the input of the override operation, and a vehicle control unit configured to automatically start the autonomous driving control which is based on the second travel plan after the autonomous driving control is released and when the autonomous driving restore determination unit determines to restore the autonomous driving control.