A method and system for controlling a playback experience for one or more videos is disclosed. Actions are specified in control documents for the one or more videos. The actions specify start time and duration for each action, optional introductory or confirmation messages or interface controls, optional gestures, and/or intents that are triggered by the actions. The various control documents are compiled into a single control document that includes a link to the one or more videos and the actions various. Multiple parties can control the playback experience with multiple control documents to provide a multi-layered control experience.