Implementations generally relate to audio-video analytics for simulation-based training. In some implementations, a method includes obtaining a video. The method further includes detecting one or more observed actions of a user in the video. The method further includes matching the one or more observed actions with one or more predetermined key actions. The method further includes annotating the video with annotations based on the matching of the one or more observed actions with the one or more predetermined key actions.