A video summary device may determine, based on video data, a plurality of events that occurred during a period of time. The video summary device may determine first measures of relevance of a plurality of portions of the period of time. A first measure of relevance of a portion of time may be determined based on one or more events of the plurality of events. The video summary device may determine second measures of relevance of a plurality of ranges of time between different portions of time of the plurality of portions of the period of time. The second measures of relevance is determined based on the first measures of relevance determined for the different portions of time. The video summary device may determine particular frames of the video data, as a video summary of the video data, based on the second measures of relevance.