An annotation system that provides annotations for a corresponding set of entities with regard to a user is presented. The annotation system annotates the entities independently of the selection of the entities to be annotated. Upon receiving a set of entities to annotate, the annotation system identifies a set of annotation relationships between the user and the entity. Each annotation relationship identifies a reason or basis for which the entity may be relevant to the user. The annotation relationships are scored to identify the strength of the relationship between the user and the entity. A configurable threshold number of annotation relationships with the greatest affinity between the user and the entity are selected as the annotation for the entity.