Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for intelligently cancelling user inputs. In one aspect, a requests input by a user is received by a dialog engine. A prompt or notification regarding the request is output by the dialog engine. That the user has taken an action in response to the prompt or notification is determined by the dialog engine. Based on the action taken by the user, that the response corresponds to a potential cancellation command is determined by the dialog system.