A computer implemented method of generating at least one highlight video from an input video, comprising, using at least one processor for: identifying a plurality of significant frames of the input video, computing video-level features of the input video, selecting a plurality of subsets of the plurality of significant frames, for each subset, computing a similarity score indicating similarity between visual features of the subset and video-level features of the input video, clustering the input video into a plurality of clusters of sequential frames according to sequential positions within the video based on the similarity scores correlated with the plurality of significant frames, and creating at least one highlight video by selecting a cluster of sequential frames of the input video.