A cognitive assignment engine (CAE) system attempts to infer semantic meaning from textual content of an incoming message in order to use the inferred meaning to assign the message to an appropriate responder. If the message contains insufficient textual content, the system identifies ontological structures comprised by the message's graphical content and classifies each structure as a function of the structure's location within the graphical content or of an intrinsic characteristic of the structure. The system then generates a message identifier by performing a computation on these classifications and uses the identifier to retrieve a previously stored graphical template that comprises ontological structures similar to those of the incoming message. The system associates the incoming message with a semantic meaning previously associated with the template, enabling the system to classify the message and to assign the message to the correct responder.