A mobile handset device collects sensor data about the physiological state of the user of the handset. The mobile handset device receives multimedia content, which is consumed on the mobile handset. In a deployment phase, the sensor data is used to classify the user's emotional response to individual pieces of media content consumed on the mobile device. A classification model built in a training phase may be used to map sensor data to classification labels indicative of the user's emotional response to multimedia.