Systems and processes for operating an intelligent automated assistant are provided. For example, a first speech input directed to a digital assistant is received from a user. A first response is provided based on the first speech input. A session window is initiated, wherein the session window is associated with a variable speech threshold. A second speech input is received during the session window. In accordance with a determination that the second speech input includes speech directed to the digital assistant, a duration associated with the session window is increased. In accordance with a determination that the variable speech threshold does not exceed a predetermined speech threshold, the session window is ended.