ET-CycleGAN: Generating thermal images from images in the visible spectrum for facial emotion recognition

Gerard Pons, Abdallah El Ali, Pablo Cesar

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

38 Downloads (Pure)


Facial thermal imaging has in recent years shown to be an efficient modality for facial emotion recognition. However, the use of deep learning in this field is still not fully exploited given the small number and size of the current datasets. The goal of this work is to improve the performance of the existing deep networks in thermal facial emotion recognition by generating new synthesized thermal images from images in the visual spectrum (RGB). To address this challenging problem, we propose an emotion-guided thermal CycleGAN (ET-CycleGAN). This Generative Adversarial Network (GAN) regularizes the training with facial and emotion priors by extracting features from Convolutional Neural Networks (CNNs) trained for face recognition and facial emotion recognition, respectively. To assess this approach, we generated synthesized images from the training set of the USTC-NVIE dataset, and included the new data to the training set as a data augmentation strategy. By including images generated using the ET-CycleGAN, the accuracy for emotion recognition increased by 10.9%. Our initial findings highlight the importance of adding priors related to training set image attributes (in our case face and emotion priors), to ensure such attributes are maintained in the generated images.

Original languageEnglish
Title of host publicationICMI 2020 Companion
Subtitle of host publicationCompanion Publication of the 2020 International Conference on Multimodal Interaction
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
Number of pages5
ISBN (Print)978-1-4503-8002-7
Publication statusPublished - 2020
Event2020 International Conference on Multimodal Interaction, ICMI 2020 - Virtual, Online, Netherlands
Duration: 25 Oct 202029 Oct 2020


Conference2020 International Conference on Multimodal Interaction, ICMI 2020
CityVirtual, Online

Bibliographical note

Accepted author manuscript


  • Emotion recognition
  • Generative adversarial networks
  • Thermal imaging


Dive into the research topics of 'ET-CycleGAN: Generating thermal images from images in the visible spectrum for facial emotion recognition'. Together they form a unique fingerprint.

Cite this