Collaborator content selection and manipulation actions and comments during interaction with a document are recorded to enhance asynchronous collaboration. The collaborator content selection and manipulation actions and comments are recorded during interaction with a document in a first collaboration viewport at a first time and played back in a second collaboration viewport at a second time while the same or a different instance of the document is displayed. The collaborator content selection and manipulation actions and comments include at least text selection, mouse or pen movements, and voice comments or ink annotations that are played back to recreate the collaborator changes and comments as though they were occurring in real-time, thereby creating a user experience that is more like the “over the shoulder” collaboration that would be available to collaborators that are at the same physical location.