Collaborative annotation of a document. A GUI is generated by a host computer that displays a document to annotate. The GUI is transmitted by the host computer to a plurality of client computers. Real time communication modes are established by the host computer between the client computers, each including communication channels between all the client computers. The host computer receives annotations entered into the GUIs at the client computers, each annotation identifying a span of words in the document to annotate and including an associated annotation type. The host computer updates the GUI to include for display the received annotations and inter-rater agreement statistics indicating the level of agreement between annotations received from each client computer and annotations received from the other client computers. The updated GUI is then transmitted by the host computer to the plurality of client computers.