A system and method for filtering email annotations is disclosed. An email may be parsed for actionable items and email text associated with those items may be annotated. A server-side filter may be applied to the actionable items to remove annotations that have a low-probability for user action. The email and its associated annotations are then stored on the server. An email client may retrieve the email and its associated annotations from the server and perform render-time filtering. This render-time filtering applies real-time checking for annotations which have a low-probability for user action. The email and its associated annotations are then displayed via a user interface.