A method is provided to automatically determine “exciting” segments from a video. The method includes calculating image features of each frame in the video, determining a difference for each pair of adjacent frames, calculating a sum of differences for each group of frames in the video, and selecting a number of the groups with high sums as exciting segments of the video. The differences between pairs of adjacent frames are used as a criterion for measuring a degree of “excitement” for determining the highlights in the video.