A speech-based system is configured to interact with a user through speech to determine intents and goals of the user. The system may analyze multiple dialog turns in order to determine and fully define a goal that the user is trying to express. Each dialog turn comprises a user utterance. Each dialog turn may also comprise a system speech response. In order to evaluate the performance of the system, logged data is analyzed to identify goal segments within the logged data, where a goal segment is a sequence of dialog turns that relate to a corresponding user goal. A subset of the dialog turns is annotated manually to delineate goal segments. A predictive model is then constructed based on the manually annotated goal segments. The predictive model is then used to identify goal segments formed by additional dialog turns.