This disclosure relates to systems and methods that generate highlights for a video. A video may be accessed. Criteria for identifying a moment of interest within the video based on a user interaction with a portion of the video may be obtained. Interaction information indicating the user interaction with the portion of the video may be received. The interaction information for the portion of the video may be compared with the criteria. Responsive to the interaction information for the portion of the video indicating the user interaction with the portion of the video satisfying the criteria, a moment in the video corresponding to the portion of the video may be associated with the moment of interest.