Computer implemented methods and systems are provided for training a machine learning architecture for surgical performance tracking and measurement based on surgical procedure video data set. The methods and systems include, in a first aspect, a sequential relation architecture and a dimensionality reduction architecture. In a second aspect, the methods and systems include a surgical instrument instance segmentation architecture, a decomposition model, and a sequential relation architecture. The video data is processed on a frame level to generate compressed or reduced representations of the video data.