A system and method for incorporating motion into a speech processing system. A wearable device that is capable of both capturing spoken utterances and capturing motion data may be used to interact with a speech processing system. In certain circumstances, such as when voice communication are unreliable (due to noise) or when controlling the system by motion is desired, motion of a device may be used to provide input to a speech processing system. For example, sensor data or gesture data resulting from movement of a device may be processed and input into a natural language system as representative of a spoken command portion or other input. The motion information may be interpreted to provide prompts to the system (e.g., “yes,” “no,” etc.), to perform certain commands (skip, forward, back, cancel) or to otherwise control the system.