One embodiment of the present invention provides a system for tracking and distributing annotations for a video stream. During operation, the system receives, at an annotation server, the video stream originating from a remote field computer, extracts a number of features from the received video stream, and identifies a group of features that matches a known feature group, which is associated with an annotation. The system further associates the identified group of features with the annotation, and forwards the identified group of features and the annotation to the remote field computer, thereby facilitating the remote field computer to associate the annotation with a group of locally extracted features and display the video stream with the annotation placed in a location based at least on locations of the locally extracted features.