On Social Involvement in Mingling Scenarios: Detecting Associates of F-formations in Still Images

Lu Zhang, Hayley Hung

Research output: Contribution to journalArticleScientificpeer-review

3 Citations (Scopus)
105 Downloads (Pure)


In this paper, we carry out an extensive study of social involvement in free standing conversing groups (the so-called F-formations) from static images. By introducing a novel feature representation, we show that the standard features which have been used to represent full membership in an F-formation cannot be applied to the detection of so-called associates of F-formations due to their sparser nature. We also enrich state-of-The-Art F-formation modelling by learning a frustum of attention that accounts for the spatial context. That is, F-formation configurations vary with respect to the arrangement of furniture and the non-uniform crowdedness in the space during mingling scenarios. Moroever, the majority of prior works have considered the labelling of conversing groups as an objective task, requiring only a single annotator. However, we show that by embracing the subjectivity of social involvement, we not only generate a richer model of the social interactions in a scene but can use the detected associates to improve initial estimates of the full members of an F-formation. We carry out extensive experimental validation of our proposed approach by collecting a novel set of multi-Annotator labels of involvement on two publicly available datasets; The Idiap Poster Data and SALSA data set. Moreover, we show that parameters learned from the Idiap Poster Data can be transferred to the SALSA data, showing the power of our proposed representation in generalising over new unseen data from a different environment.

Original languageEnglish
Article number8413103
Pages (from-to)165-176
Number of pages12
JournalIEEE Transactions on Affective Computing
Issue number1
Publication statusPublished - 2020

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.


  • F-formations detection
  • human behaviour analysis
  • social group detection


Dive into the research topics of 'On Social Involvement in Mingling Scenarios: Detecting Associates of F-formations in Still Images'. Together they form a unique fingerprint.

Cite this