The subject disclosure is generally directed towards synchronizing live streaming videos with additional sources of contextually-related data. In one or more aspects, contextually-related data is marked with a timestamp, with a reference to each piece of data maintained in a manifest. The manifest is accessed to locate the contextually relevant information (e.g., on a companion device or the video playing device) in synchronization with the streaming video, such that a user is able to pause/stop/rewind/fast forward while maintaining the timeline synchronization between video and the data.