The present invention describes a system and method for controlling interactive video, including a remote control device allowing the user to interact with the video. The method allows bringing the “lean-forward” hypervideo experience to a “lay-back” environment such as the TV screen. The method also allows for the creation of a single system that combines both the social aspects of the “lay-back” and the “lean-forward experience” into a new type of rich media viewing experience.