Embodiments are directed towards providing supplemental content to a viewer based on the viewer's emotion and environment while the viewer is viewing content. Content is provided to a content receiver for presentation to the viewer. While the viewer is viewing the content, an image of the viewer is captured and analyzed to detect the viewer's emotion and environment. Supplemental content is selected based on this detected viewer emotion and environment and presented to the viewer, which allows the supplemental content to be tailored to the emotional state of the viewer. One or more additional images of the viewer are captured while the viewer is viewing the supplemental content and the viewer's emotion while viewing the supplemental content is detected. This subsequent viewer emotion is analyzed to determine if the viewer had an expected response to the supplemental content, which can be used to determine the efficiency of the supplemental content.