Just the right mood for HIT!: Analyzing the role of worker moods in conversational microtask crowdsourcing

Sihang Qiu; Ujwal Gadiraju; Alessandro Bozzon

doi:10.1007/978-3-030-50578-3_26

Just the right mood for HIT! Analyzing the role of worker moods in conversational microtask crowdsourcing

Sihang Qiu^*, Ujwal Gadiraju, Alessandro Bozzon

^*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

6 Citations (Scopus)

7 Downloads (Pure)

Abstract

Conversational agents are playing an increasingly important role in providing users with natural communication environments, improving outcomes in a variety of domains in human-computer interaction. Crowdsourcing marketplaces are simultaneously flourishing, and it has never been easier to acquire large-scale human input from online workers. Recent works have revealed the potential of conversational interfaces in improving worker engagement and satisfaction. At the same time, worker moods have been shown to have significant effects on quality related outcomes. Little is known about the role of worker moods in shaping work in conversational microtask crowdsourcing. In this paper, we conducted a crowdsourcing study addressing 600 unique online workers, to investigate the role that worker moods play in conversational microtask crowdsourcing. We also explore whether suitable conversational styles of the agent can affect the performance of workers in different moods. Our results show that workers in a pleasant mood tend to produce significantly higher quality results (over 20%), exhibit greater engagement (an increase by around 19%) and report a lower cognitive load (by over 12%), and a suitable conversational style can have a significant impact on workers in different moods. Our findings advance the current understanding of conversational microtask crowdsourcing and have important implications on designing future conversational crowdsourcing systems.

Original language	English
Title of host publication	Web Engineering - 20th International Conference, ICWE 2020, Proceedings
Editors	Maria Bielikova, Tommi Mikkonen, Cesare Pautasso
Place of Publication	Cham
Publisher	Springer Science+Business Media
Pages	381-396
Number of pages	16
ISBN (Electronic)	978-3-030-50578-3
ISBN (Print)	978-3-030-50577-6
DOIs	https://doi.org/10.1007/978-3-030-50578-3_26
Publication status	Published - 2020
Event	20th International Conference on Web Engineering, ICWE 2020 - Helsinki, Finland Duration: 9 Jun 2020 → 12 Jun 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12128
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	20th International Conference on Web Engineering, ICWE 2020
Country/Territory	Finland
City	Helsinki
Period	9/06/20 → 12/06/20

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

Conversational agent
Conversational style
Crowdsourcing
Moods
Worker moods
Worker performance

Access to Document

10.1007/978-3-030-50578-3_26

Just_the_Right_Mood_for_HIT_Final published version, 1.06 MB

Cite this

Qiu, S., Gadiraju, U., & Bozzon, A. (2020). Just the right mood for HIT! Analyzing the role of worker moods in conversational microtask crowdsourcing. In M. Bielikova, T. Mikkonen, & C. Pautasso (Eds.), Web Engineering - 20th International Conference, ICWE 2020, Proceedings (pp. 381-396). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12128 ). Springer Science+Business Media. https://doi.org/10.1007/978-3-030-50578-3_26

Qiu, Sihang ; Gadiraju, Ujwal ; Bozzon, Alessandro. / Just the right mood for HIT! Analyzing the role of worker moods in conversational microtask crowdsourcing. Web Engineering - 20th International Conference, ICWE 2020, Proceedings. editor / Maria Bielikova ; Tommi Mikkonen ; Cesare Pautasso. Cham : Springer Science+Business Media, 2020. pp. 381-396 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{4a12dc1642e64cfb97ca47bbe30f130e,

title = "Just the right mood for HIT!: Analyzing the role of worker moods in conversational microtask crowdsourcing",

abstract = "Conversational agents are playing an increasingly important role in providing users with natural communication environments, improving outcomes in a variety of domains in human-computer interaction. Crowdsourcing marketplaces are simultaneously flourishing, and it has never been easier to acquire large-scale human input from online workers. Recent works have revealed the potential of conversational interfaces in improving worker engagement and satisfaction. At the same time, worker moods have been shown to have significant effects on quality related outcomes. Little is known about the role of worker moods in shaping work in conversational microtask crowdsourcing. In this paper, we conducted a crowdsourcing study addressing 600 unique online workers, to investigate the role that worker moods play in conversational microtask crowdsourcing. We also explore whether suitable conversational styles of the agent can affect the performance of workers in different moods. Our results show that workers in a pleasant mood tend to produce significantly higher quality results (over 20%), exhibit greater engagement (an increase by around 19%) and report a lower cognitive load (by over 12%), and a suitable conversational style can have a significant impact on workers in different moods. Our findings advance the current understanding of conversational microtask crowdsourcing and have important implications on designing future conversational crowdsourcing systems.",

keywords = "Conversational agent, Conversational style, Crowdsourcing, Moods, Worker moods, Worker performance",

author = "Sihang Qiu and Ujwal Gadiraju and Alessandro Bozzon",

note = "Green Open Access added to TU Delft Institutional Repository {\textquoteleft}You share, we take care!{\textquoteright} – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. ; 20th International Conference on Web Engineering, ICWE 2020 ; Conference date: 09-06-2020 Through 12-06-2020",

year = "2020",

doi = "10.1007/978-3-030-50578-3_26",

language = "English",

isbn = "978-3-030-50577-6",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science+Business Media",

pages = "381--396",

editor = "Maria Bielikova and Tommi Mikkonen and Cesare Pautasso",

booktitle = "Web Engineering - 20th International Conference, ICWE 2020, Proceedings",

}

Qiu, S, Gadiraju, U & Bozzon, A 2020, Just the right mood for HIT! Analyzing the role of worker moods in conversational microtask crowdsourcing. in M Bielikova, T Mikkonen & C Pautasso (eds), Web Engineering - 20th International Conference, ICWE 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12128 , Springer Science+Business Media, Cham, pp. 381-396, 20th International Conference on Web Engineering, ICWE 2020, Helsinki, Finland, 9/06/20. https://doi.org/10.1007/978-3-030-50578-3_26

Just the right mood for HIT! Analyzing the role of worker moods in conversational microtask crowdsourcing. / Qiu, Sihang; Gadiraju, Ujwal ; Bozzon, Alessandro.

Web Engineering - 20th International Conference, ICWE 2020, Proceedings. ed. / Maria Bielikova; Tommi Mikkonen; Cesare Pautasso. Cham : Springer Science+Business Media, 2020. p. 381-396 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12128 ).

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Just the right mood for HIT!

T2 - 20th International Conference on Web Engineering, ICWE 2020

AU - Qiu, Sihang

AU - Gadiraju, Ujwal

AU - Bozzon, Alessandro

N1 - Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2020

Y1 - 2020

N2 - Conversational agents are playing an increasingly important role in providing users with natural communication environments, improving outcomes in a variety of domains in human-computer interaction. Crowdsourcing marketplaces are simultaneously flourishing, and it has never been easier to acquire large-scale human input from online workers. Recent works have revealed the potential of conversational interfaces in improving worker engagement and satisfaction. At the same time, worker moods have been shown to have significant effects on quality related outcomes. Little is known about the role of worker moods in shaping work in conversational microtask crowdsourcing. In this paper, we conducted a crowdsourcing study addressing 600 unique online workers, to investigate the role that worker moods play in conversational microtask crowdsourcing. We also explore whether suitable conversational styles of the agent can affect the performance of workers in different moods. Our results show that workers in a pleasant mood tend to produce significantly higher quality results (over 20%), exhibit greater engagement (an increase by around 19%) and report a lower cognitive load (by over 12%), and a suitable conversational style can have a significant impact on workers in different moods. Our findings advance the current understanding of conversational microtask crowdsourcing and have important implications on designing future conversational crowdsourcing systems.

AB - Conversational agents are playing an increasingly important role in providing users with natural communication environments, improving outcomes in a variety of domains in human-computer interaction. Crowdsourcing marketplaces are simultaneously flourishing, and it has never been easier to acquire large-scale human input from online workers. Recent works have revealed the potential of conversational interfaces in improving worker engagement and satisfaction. At the same time, worker moods have been shown to have significant effects on quality related outcomes. Little is known about the role of worker moods in shaping work in conversational microtask crowdsourcing. In this paper, we conducted a crowdsourcing study addressing 600 unique online workers, to investigate the role that worker moods play in conversational microtask crowdsourcing. We also explore whether suitable conversational styles of the agent can affect the performance of workers in different moods. Our results show that workers in a pleasant mood tend to produce significantly higher quality results (over 20%), exhibit greater engagement (an increase by around 19%) and report a lower cognitive load (by over 12%), and a suitable conversational style can have a significant impact on workers in different moods. Our findings advance the current understanding of conversational microtask crowdsourcing and have important implications on designing future conversational crowdsourcing systems.

KW - Conversational agent

KW - Conversational style

KW - Crowdsourcing

KW - Moods

KW - Worker moods

KW - Worker performance

UR - http://www.scopus.com/inward/record.url?scp=85087041312&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-50578-3_26

DO - 10.1007/978-3-030-50578-3_26

M3 - Conference contribution

AN - SCOPUS:85087041312

SN - 978-3-030-50577-6

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 381

EP - 396

BT - Web Engineering - 20th International Conference, ICWE 2020, Proceedings

A2 - Bielikova, Maria

A2 - Mikkonen, Tommi

A2 - Pautasso, Cesare

PB - Springer Science+Business Media

CY - Cham

Y2 - 9 June 2020 through 12 June 2020

ER -

Qiu S, Gadiraju U , Bozzon A. Just the right mood for HIT! Analyzing the role of worker moods in conversational microtask crowdsourcing. In Bielikova M, Mikkonen T, Pautasso C, editors, Web Engineering - 20th International Conference, ICWE 2020, Proceedings. Cham: Springer Science+Business Media. 2020. p. 381-396. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-030-50578-3_26