Systems, methods, and computer-readable storage media are provided for recognizing emotion in audio signals in real-time. An audio signal is detected and a rapid audio fingerprint is computed on a user's computing device. One or more features is extracted from the audio fingerprint and compared with features associated with defined emotions to determine relative degrees of similarity. Confidence scores are computed for the defined emotions based on the relative degrees of similarity and it is determined whether a confidence score for one or more particular emotions exceeds a threshold confidence score. If it is determined that a threshold confidence score for one or more particular emotions is exceeded, the particular emotion or emotions are associated with the audio signal. As desired, various action then may be initiated based upon the emotion/emotions associated with the audio signal.