An intent confusion evaluation engine receives conversation data corresponding to conversations between customers and agents. The engine evaluates annotations in the conversation data corresponding to intents identified from messages exchanged between customers and agents to determine levels of confusion amongst different intents. Based on these levels of confusion, the engine creates a graphical representation that illustrates the various intents and the level of confusion between different pairings of intents for the set of conversations. If an update is provided to the annotations, the graphical representation is updated dynamically and in real-time to provide updated levels of confusion amongst the various intents in accordance with the update.