Embodiments are directed to summarizing conversational speech. Conversation segments may be provided based on a conversation stream and segmentation models. Summarization models may be determined based on characteristics of the conversation segments. Summarization information may be generated for each of the conversation segments based on the summarization models such that the summarization information includes a text-based summarization of the conversation segment. Summarization profiles may be generated for the conversation segments based on the summarization information such that each summarization profile is associated with quality scores. Summarization models may be modified based on the summarization profiles and the associated quality scores such that the summarization profiles are updated based on the modified summarization models. Modified summarization models and the updated summarization profiles may be employed to provide reports to a user.