A method includes using a computing platform to extract audio and video segments from current video data of a person; apply a first trained supervised machine learning (ML) model to the video segment to produce a first probability distribution of emotional states of the person; and apply a second trained supervised ML model to the audio segment to produce a second probability distribution of the emotional states of the person. The first and second probability distributions are used by the computing platform to produce a third probability distribution of mismatch-based emotional states of the person. The third probability distribution is based on mismatches between the first and second probability distributions.