Techniques for ranking representative segments in media data are provided. Media features of many different types may be extracted from the media data. A plurality of ranking scores may be assigned to a plurality of candidate representative segments. Each individual candidate representative segment in the plurality of candidate representative segments comprises at least one scene in one or more statistical patterns in media features of the media data based on one or more types of features extractable from the media data. Each individual ranking score in the plurality of ranking scores may be assigned to an individual candidate representative segment in the plurality of candidate representative segments. A representative segment to be played to an end user may be selected from the candidate representative segments, based on the plurality of ranking scores.