Described herein are systems and methods for detecting a physiological response based on multispectral data. In one embodiment, a system includes an inward-facing head-mounted thermal camera (CAM) that takes thermal measurements of a first region of interest (THROI1) on a user's face, and an inward-facing head-mounted visible-light camera (VCAM) that takes images of a second region of interest (IMROI2) on the face. The first and second regions of interest overlap, and the system includes a computer that detects the physiological response based on THROI1, IMROI2, and a model. Optionally, the model was trained based on previous THROI1 and IMROI2 of the user taken during different days. Optionally, the physiological response is indicative of an occurrence of an emotional state of the user, such as joy, fear, sadness or anger.