Impact of Annotation Modality on Label Quality and Model Performance in the Automatic Assessment of Laughter In-the-Wild

Research output: Contribution to journalArticleScientificpeer-review


Although laughter is known to be a multimodal signal, it is primarily annotated from audio. It is unclear how laughter labels may differ when annotated from modalities like video, which capture body movements and are relevant in in-the-wild studies. In this work we ask whether annotations of laughter are congruent across modalities, and compare the effect that labeling modality has on machine learning model performance. We compare annotations and models for laughter detection, intensity estimation, and segmentation, using a challenging in-the-wild conversational dataset with a variety of camera angles, noise conditions and voices. Our study with 48 annotators revealed evidence for incongruity in the perception of laughter and its intensity between modalities, mainly due to lower recall in the video condition. Our machine learning experiments compared the performance of modern unimodal and multi-modal models for different combinations of input modalities, training, and testing label modalities. In addition to the same input modalities rated by annotators (audio and video), we trained models with body acceleration inputs, robust to cross-contamination, occlusion and perspective differences. Our results show that performance of models with body movement inputs does not suffer when trained with video-acquired labels, despite their lower inter-rater agreement.

Original languageEnglish
Pages (from-to)1-17
Number of pages17
JournalIEEE Transactions on Affective Computing
Publication statusE-pub ahead of print - 2023


  • Action recognition
  • annotation
  • Annotations
  • Cameras
  • continuous annotation
  • Face recognition
  • Labeling
  • laughter
  • laughter detection
  • laughter intensity
  • Machine learning
  • mingling datasets
  • Physiology
  • Task analysis


Dive into the research topics of 'Impact of Annotation Modality on Label Quality and Model Performance in the Automatic Assessment of Laughter In-the-Wild'. Together they form a unique fingerprint.

Cite this