Computer vision systems for segmenting scenes into semantic components identify a differential within the physiological readings from the user. The differential corresponds to a semantic boundary associated with the user's gaze. Based upon data gathered by a gaze tracking device, the computer vision system identifies a relative location of the user's gaze at the time of the identified differential. The computer vision system then associates the relative location of the user's gaze with a semantic boundary.