Provided are systems, methods, and media for handling dialogs based on user behavior data. An example method includes receiving an input paragraph having one or more factual sentences, in which each of the one or more factual sentences includes one or more words. Receiving an input question comprising one or more words. Performing word-level gaze prediction on the input paragraph to identify one or more predicted gaze attributes for the input paragraph. Extracting an answer to the input question based, at least in part, on the input paragraph, the input question, and the one or more predicted gaze attributes of the input paragraph. Transmitting the extracted answer.