Techniques are described for analyzing communications sent during a service session to provide (e.g., customer) service on a social media channel, the analysis to determine a quality of service provided during the session. Natural language processing, lexical analysis, pattern matching, or other types of analysis may be used to determine an empathy factor and a conversational factor for communications between a service representative (SR) and a user during a session. The factors may be combined with other factors, such as a timely acknowledgement factor and a timely resolution factor, to generate a response quality index (RQI) for the session. Based on the RQI, feedback information may be generated and sent to the SR. In some implementations, the communications may be analyzed and feedback information sent to the SR in real time during the session, to dynamically improve service quality.