Embodiments described herein provide various examples of automatically processing surgical videos to detect surgical tools and tool-related events, and extract surgical-tool usage information. In one aspect, a process for automatically detecting a new surgical tool engagement during a recorded surgical procedure is disclosed. This process can begin by receiving a surgical procedure video and then segmenting the surgical video into sequences of video frames. Next, for each sequence of video frames, the video frames are processed to detect one or more surgical tools and one or more surgical tool engagements associated with the detected surgical tools. If a surgical tool engagement is detected in the sequence of video frames, the process then determines if a detected surgical tool associated with the detected surgical tool engagement is associated with a previously identified surgical tool engagement. If not, the process identifies the detected surgical tool engagement as a new surgical tool engagement.