Few-shot Learning for Fine-grained Emotion Recognition using Physiological Signals

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Fine-grained emotion recognition can model the temporal dynamics of emotions. It is temporally more precise when compared to predicting one emotion for activities (e.g., video clip watching). Previous works require large amounts of continuously annotated data to train an accurate recognition model. However, the experiments to collect large amounts of continuously annotated physiological signals are costly and time-consuming. To overcome this challenge, we propose a few-shot learning algorithm EmoDSN which can rapidly converge on a small amount of training data (typically < 10 samples per class (i.e., < 10 shot)) for fine-grained emotion recognition. EmoDSN recognizes fine-grained valence and arousal (V-A) labels by maximizing the distance metric between signal segments with different V-A labels. We tested EmoDSN on three different datasets, CASE, MERCA and CEAP-360VR, collected in three different environments: desktop, mobile and HMD-based virtual reality, respectively. The results from our experiments show that EmoDSN achieves promising results for both one-dimension binary (high/low V-A, 1D-2C) and two-dimensional 5-class (four quadrants of V-A space + neutral, 2D-5C) classification. We get an averaged accuracy of 76.04%, 76.62% and 57.62% for 1D-2C valence, 1D-2C arousal and 2D-5C respectively by using only 5 shot of training data. We also find that EmoDSN can achieve better recognition results trained with fewer annotated samples if we select training samples from the changing points of emotion and the ending moments of video watching.

Original languageEnglish
Article number9751421
JournalIEEE Transactions on Multimedia
DOIs
Publication statusAccepted/In press - 2022

Keywords

  • emotion recognition
  • deep siamese network
  • physiological signals
  • small data

Fingerprint

Dive into the research topics of 'Few-shot Learning for Fine-grained Emotion Recognition using Physiological Signals'. Together they form a unique fingerprint.

Cite this