An information handling system (IHS) controls a display for an immersive experience based upon a gazing view of one or more users. A processor communicates with the display and a gaze detector to detect a relative location of the user(s) to the display. A graphic processing unit (GPU) allocates a field of view on the display based upon the relative location, centers display content in the field of view on the display, and accesses color point control data in response to the allocated field of view. The GPU manages color and brightness of display components of the display according to the accessed color point control data to compensate for variations of color and brightness in a viewing angle of the field of view.