An autonomous driving control device is capable of starting an autonomous driving control without an operation of a driver and reducing a possibility that the driver can not start manual driving. An autonomous driving control is switched to manual driving when a determination section determines that the amount of operation by the driver is equal to or greater than a first threshold, before a predetermined time elapses since the autonomous driving control is automatically started. An autonomous driving control is switched to a manual driving when the determination section determines that the amount of operation by the driver is equal to or greater than a second threshold that is greater than the first threshold, after the predetermined time elapses.