An autonomous driving system includes an electronic control unit configured to generate, based on a target route, a target path in a predetermined coordinate system and a speed plan specifying a passage time at a control point on the target path, and rebuild, based on an actual speed of a vehicle, the speed plan when an operation intervention is performed during autonomous driving performed by autonomous driving control for causing the vehicle to travel along the target path according to the speed plan, the operation intervention changing a braking force acting on the vehicle, and perform the autonomous driving control by using an actuator mounted on the vehicle.