In one embodiment, a driving environment is perceived based on sensor data obtained from a variety of sensors, including determining a current speed of an ADV. In response to a request for driving with an idle speed, a speed guideline is generated based on an idle speed curve in view of the current speed of the ADV. A speed planning operation is performed by optimizing a cost function based on the speed guideline to determine the speeds of the trajectory points at different points in time along a trajectory planned to drive the ADV. One or more control commands are then generated to control the ADV with the planned speeds along the planned trajectory, such that the ADV moves according to an intended idle speed.