An autonomous driving system includes a lane change control device that performs lane change control for making a lane change from a first lane to a second lane during autonomous driving of a vehicle. From start to completion of the lane change control, the lane change control device determines whether or not a driver's operation is performed as an abort request operation that requests to abort the lane change control. Specifically, the lane change control device: calculates abort necessity level indicating necessity to abort the lane change control, based on driving environment information indicating driving environment for the vehicle; sets criterion for determination such that the criterion becomes more easily satisfied as the abort necessity level increases; and when the driver's operation satisfies the criterion for determination, determines that the driver's operation is performed as the abort request operation.