Technologies are described for enabling interactivity with video games by viewers of a live video game streaming service. For example, messages can be received indicating interactive control events initiated by remote viewers of a live interactive video game stream. A control state data structure can be maintained that comprises control state for the received interactive control events. The control state data structure can maintain separate control state for each of a plurality of time periods (e.g., each time period corresponding to one or more video frames). Actions in the video game can be activated based on the control state data structure.