Detecting F-formations Roles in Crowded Social Scenes with Wearables: Combining Proxemics Dynamics using LSTMs

Alessio Rosatelli; Ekin Gedik; Hayley Hung

doi:10.1109/ACIIW.2019.8925179

Detecting F-formations Roles in Crowded Social Scenes with Wearables: Combining Proxemics Dynamics using LSTMs

Alessio Rosatelli, Ekin Gedik, Hayley Hung

Pattern Recognition and Bioinformatics

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

7 Citations (Scopus)

10 Downloads (Pure)

Abstract

In this paper, we investigate the use of proxemics and dynamics for automatically identifying conversing groups, or so-called F-formations. More formally we aim to automatically identify whether wearable sensor data coming from 2 people is indicative of F-formation membership. We also explore the problem of jointly detecting membership and more descriptive information about the pair relating to the role they take in the conversation (i.e. speaker or listener). We jointly model the concepts of proxemics and dynamics using binary proximity and acceleration obtained through a single wearable sensor per person. We test our approaches on the publicly available MatchNMingle dataset which was collected during real-life mingling events. We find out that fusion of these two modalities performs significantly better than them independently, providing an AUC of 0.975 when data from 30-second windows are used. Furthermore, our investigation into roles detection shows that each role pair requires a different time resolution for accurate detection.

Original language	English
Title of host publication	2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)
Publisher	IEEE
Pages	147-153
Number of pages	7
ISBN (Electronic)	978-1-7281-3891-6
ISBN (Print)	978-1-7281-3892-3
DOIs	https://doi.org/10.1109/ACIIW.2019.8925179
Publication status	Published - 2019
Event	8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos, ACIIW 2019 - Cambridge, United Kingdom Duration: 3 Sept 2019 → 6 Sept 2019

Conference

Conference	8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos, ACIIW 2019
Country/Territory	United Kingdom
City	Cambridge
Period	3/09/19 → 6/09/19

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

conversing groups
F-formation detection
recurrent neural networks
role identification
wearable sensing

Access to Document

10.1109/ACIIW.2019.8925179

8925179Final published version, 656 KB

Cite this

@inproceedings{be1f228f3bf1437ca6ed9a642e627e2d,

title = "Detecting F-formations Roles in Crowded Social Scenes with Wearables: Combining Proxemics Dynamics using LSTMs",

abstract = "In this paper, we investigate the use of proxemics and dynamics for automatically identifying conversing groups, or so-called F-formations. More formally we aim to automatically identify whether wearable sensor data coming from 2 people is indicative of F-formation membership. We also explore the problem of jointly detecting membership and more descriptive information about the pair relating to the role they take in the conversation (i.e. speaker or listener). We jointly model the concepts of proxemics and dynamics using binary proximity and acceleration obtained through a single wearable sensor per person. We test our approaches on the publicly available MatchNMingle dataset which was collected during real-life mingling events. We find out that fusion of these two modalities performs significantly better than them independently, providing an AUC of 0.975 when data from 30-second windows are used. Furthermore, our investigation into roles detection shows that each role pair requires a different time resolution for accurate detection.",

keywords = "conversing groups, F-formation detection, recurrent neural networks, role identification, wearable sensing",

author = "Alessio Rosatelli and Ekin Gedik and Hayley Hung",

note = "Green Open Access added to TU Delft Institutional Repository {\textquoteleft}You share, we take care!{\textquoteright} – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. ; 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos, ACIIW 2019 ; Conference date: 03-09-2019 Through 06-09-2019",

year = "2019",

doi = "10.1109/ACIIW.2019.8925179",

language = "English",

isbn = "978-1-7281-3892-3",

pages = "147--153",

booktitle = "2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)",

publisher = "IEEE",

address = "United States",

}

Rosatelli, A, Gedik, E & Hung, H 2019, Detecting F-formations Roles in Crowded Social Scenes with Wearables: Combining Proxemics Dynamics using LSTMs. in 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)., 8925179, IEEE, pp. 147-153, 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos, ACIIW 2019, Cambridge, United Kingdom, 3/09/19. https://doi.org/10.1109/ACIIW.2019.8925179

Detecting F-formations Roles in Crowded Social Scenes with Wearables: Combining Proxemics Dynamics using LSTMs. / Rosatelli, Alessio; Gedik, Ekin; Hung, Hayley.
2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 2019. p. 147-153 8925179.

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Detecting F-formations Roles in Crowded Social Scenes with Wearables

T2 - 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos, ACIIW 2019

AU - Rosatelli, Alessio

AU - Gedik, Ekin

AU - Hung, Hayley

N1 - Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2019

Y1 - 2019

N2 - In this paper, we investigate the use of proxemics and dynamics for automatically identifying conversing groups, or so-called F-formations. More formally we aim to automatically identify whether wearable sensor data coming from 2 people is indicative of F-formation membership. We also explore the problem of jointly detecting membership and more descriptive information about the pair relating to the role they take in the conversation (i.e. speaker or listener). We jointly model the concepts of proxemics and dynamics using binary proximity and acceleration obtained through a single wearable sensor per person. We test our approaches on the publicly available MatchNMingle dataset which was collected during real-life mingling events. We find out that fusion of these two modalities performs significantly better than them independently, providing an AUC of 0.975 when data from 30-second windows are used. Furthermore, our investigation into roles detection shows that each role pair requires a different time resolution for accurate detection.

AB - In this paper, we investigate the use of proxemics and dynamics for automatically identifying conversing groups, or so-called F-formations. More formally we aim to automatically identify whether wearable sensor data coming from 2 people is indicative of F-formation membership. We also explore the problem of jointly detecting membership and more descriptive information about the pair relating to the role they take in the conversation (i.e. speaker or listener). We jointly model the concepts of proxemics and dynamics using binary proximity and acceleration obtained through a single wearable sensor per person. We test our approaches on the publicly available MatchNMingle dataset which was collected during real-life mingling events. We find out that fusion of these two modalities performs significantly better than them independently, providing an AUC of 0.975 when data from 30-second windows are used. Furthermore, our investigation into roles detection shows that each role pair requires a different time resolution for accurate detection.

KW - conversing groups

KW - F-formation detection

KW - recurrent neural networks

KW - role identification

KW - wearable sensing

UR - http://www.scopus.com/inward/record.url?scp=85077822177&partnerID=8YFLogxK

U2 - 10.1109/ACIIW.2019.8925179

DO - 10.1109/ACIIW.2019.8925179

M3 - Conference contribution

SN - 978-1-7281-3892-3

SP - 147

EP - 153

BT - 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)

PB - IEEE

Y2 - 3 September 2019 through 6 September 2019

ER -

Detecting F-formations Roles in Crowded Social Scenes with Wearables: Combining Proxemics Dynamics using LSTMs

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this