A system and method provide server side support for seamless, scalable rewind and playback of a video stream. A video stream is stored and indexed in a network storage place. A video stream can be indexed at frame level where each intra frame of the video stream has an index indicating the file offset and the time stamp of the intra frame in the video stream. A user request for rewinding of a video stream while the video stream is being broadcast is processed by extracting the rewinding time requirement from the user request. The extracted rewinding time value is used to calculate the requested file offset. The video stream starting at the requested time is retrieved and played back according to the user request. The system also provides server side support for seamless rewinding of a video stream and scalable system performance across multiple streaming data centers.