Few-shot Learning for Fine-grained Emotion Recognition using Physiological Signals

Tianyi Zhang, Abdallah El Ali, Alan Hanjalic, Pablo Cesar

Research output: Contribution to journalArticleScientificpeer-review

1 Citation (Scopus)
22 Downloads (Pure)


Fine-grained emotion recognition can model the temporal dynamics of emotions, which is more precise than predicting one emotion retrospectively for an activity (e.g., video clip watching). Previous works require large amounts of continuously annotated data to train an accurate recognition model, however experiments to collect such large amounts of continuously annotated physiological signals are costly and time-consuming. To overcome this challenge, we propose an Emotion recognition algorithm based on Deep Siamese Networks (EmoDSN) which can rapidly converge on a small amount of training data, typically less than 10 samples per class (i.e., <10 shot). EmoDSN recognizes fine-grained valence and arousal (V-A) labels by maximizing the distance metric between signal segments with different V-A labels. We tested EmoDSN on three different datasets collected in three different environments: desktop, mobile and HMD-based virtual reality, respectively. The results from our experiments show that EmoDSN achieves promising results for both one-dimension binary (high/low V-A, 1D-2 C) and two-dimensional 5-class (four quadrants of V- A space + neutral, 2D-5 C) classification. We get an averaged accuracy of 76.04, 76.62 and 57.62% for 1D-2 C valence, 1D-2 C arousal, and 2D-5 C, respectively, by using only 5 shots of training data. Our experiments show that EmoDSN can achieve better results if we select training samples from the changing points of emotion or the ending moments of video watching.

Original languageEnglish
Article number9751421
Pages (from-to)3773-3787
Number of pages15
JournalIEEE Transactions on Multimedia
Publication statusE-pub ahead of print - 2022

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.


  • emotion recognition
  • deep siamese network
  • physiological signals
  • small data


Dive into the research topics of 'Few-shot Learning for Fine-grained Emotion Recognition using Physiological Signals'. Together they form a unique fingerprint.

Cite this