A system and method are provided that use point of gaze information to determine what portions of 3D media content are actually being viewed to enable a 3D media content viewing experience to be improved. Tracking eye movements of viewers to obtain such point of gaze information are used to control characteristics of the 3D media content during consumption of that media, and/or to improve or otherwise adjust or refine the 3D media content during creation thereof by a media content provider. Outputs may be generated to illustrate what in the 3D media content was viewed at incorrect depths. Such outputs may then be used in subsequent or offline analysis, e.g., by editors for media content providers when generating the 3D media itself, in order to gauge the 3D effects. A quality metric can be computed based on the point of gaze information, which can be used to analyze the interactions between viewers and the 3D media content being displayed. The quality metric may also be calibrated in order to accommodate offsets and other factors and/or to allow for aggregation of results obtained for multiple viewers.