The subject disclosure is directed towards a technology in which highlight (curated) videos are automatically generated for users to watch based upon estimated interest levels with respect to segments of a longer video or set of videos. In one aspect, statistics are collected with respect to actions of viewers of the video or set of videos. These may include playback-related actions such as the number of times a particular segment is played, and/or behavioral actions of viewers during the playback. Based upon which segments have the highest estimated interest levels, a subset of the segments are selected for a playback list, which when played provides the highlight/curated video.