A system and method is provided for managing and displaying annotations in online hosted videos. To determine related annotations associated with a video, a plurality of annotations for one or more intervals of the video are collected and organized into a plurality of groups. Each group includes annotations for intervals of the video that are similar to each other. A group having related annotations is identified, and an annotated clip of the video is formed based upon the intervals in the group. This process can be used to determine any number of groups in a given video having related annotations, forming the respective annotated intervals within a single video, and can also identify and organize annotated intervals within a large number of different videos.