A software agent, that is used to assist in providing a service, receives communications from a set of users that are attempting to use the software agent. The communications include communications that are interacting with the software agent, and communications that are not interacting with the software agent. The software agent performs natural language processing on all communications to identify such things as user sentiment, user concerns or other items in the content of the messages, and also to identify actions taken by the users in order to obtain a measure of user satisfaction with the software agent. One or more action signals are then generated based upon the identified user satisfaction with the software agent.