A vehicle control device includes the following components. An action plan generating unit generates an action plan of a vehicle. An automated driving control unit carries out a first driving mode in which at least one of acceleration/deceleration and steering of the vehicle is automatically controlled based on the action plan. An operation device accepts an operation performed by an occupant of the vehicle. A handover control unit instructs shifting to a second driving mode, in which the degree of automated driving is lower than in the first driving mode, when the occupant operates the operation device to instruct one or both of acceleration/deceleration and steering in the first driving mode. A restraint control unit generates an estimated path of the vehicle to be taken based on the operation and restrains shifting to the second driving mode if the estimated path contradicts the action plan.