A system and a method for summary collection and playing of scenes and a recording medium thereof are provided. The system includes media supply equipment, a scene description server, a scene server, and an end device. The media supply equipment supplies media data. The scene description server supplies scene description information of a scene segment corresponding to the media data, where each piece of scene description information records a playing content of the scene segment corresponding thereto. The scene server obtains the media data and a summary collect command supplied by the end device, and retrieves a plurality of partial scene segments from the media data according to an analysis result of the summary collect command and the playing content recorded by each piece of scene description information, so as to form media summary data and output the media summary data to the end device.