Non-contact Dual-modality Emotion Recognition System by CW Radar and RGB Camera

Li Zhang, Chang-Hong Fu Fu, Hong Hong, Biao Xue, Xuemei Gu, Changzhi Li, Xiaohua Zhu

Research output: Contribution to journalArticlepeer-review

3 Scopus citations


Emotion recognition has a significant impact on people's health and life quality. Recent studies have shown that emotion recognition can be achieved by analyzing audio-visual emotion channels and physiological signals. However, the main challenges are: 1. the audio-visual techniques can be easily fooled by fake facial expressions; 2. the contact physiological signal monitoring introduces additional stress and is usually unsuitable for long-term monitoring; 3. The non-contact physiological signal monitoring methods are easily affected by complex environmental conditions. In this paper, a non-contact dual-modality emotion recognition system is proposed. First, the respiratory and heartbeat signals are measured by radar and camera simultaneously. Then, a hybrid signal optimization approach is proposed to remove the influence of body motion and light conditions on the physiological signal. It includes a light-intensity-based scheme selecting proper heartbeat signal for different light conditions, and an optical-flow-based algorithm pruning signals with significant radial body motion. Finally, the features extracted from the optimized physiological signals are fused to train an emotion recognition system. As shown in the experimental results, the proposed system could achieve high classification accuracy of 89.6% for 10-fold cross-validation at sample level, and 71.0 % for cross-validation at subject level. The respiratory and heartbeat signals by non-contact approaches are demonstrated to be reliable in emotion recognition.

Original languageEnglish
Pages (from-to)23198-23212
Number of pages15
JournalIEEE Sensors Journal
Issue number20
StatePublished - Oct 15 2021


  • Dual-modality
  • Emotion recognition
  • Non-contact
  • Radar
  • Video


Dive into the research topics of 'Non-contact Dual-modality Emotion Recognition System by CW Radar and RGB Camera'. Together they form a unique fingerprint.

Cite this