TaskGenie: Crowd-Powered Task Generation for Struggling Search

Luyan Xu; Xuan Zhou; Ujwal Gadiraju

doi:10.1007/978-3-030-62008-0_1

TaskGenie: Crowd-Powered Task Generation for Struggling Search

Luyan Xu, Xuan Zhou, Ujwal Gadiraju

Web Information Systems

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

1 Citation (Scopus)

32 Downloads (Pure)

Abstract

Search tasks provide a medium for the evaluation of system performance and the underlying analytical aspects of IR systems. Researchers have recently developed new interfaces or mechanisms to support vague information needs and struggling search. However, little attention has been paid to the generation of a unified task set for evaluation and comparison of search engine improvements for struggling search. Generation of such tasks is inherently difficult, as each task is supposed to trigger struggling and exploring user behavior rather than simple search behavior. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for system evaluation. In this paper, we propose a task generation method and develop a crowd-powered platform called TaskGenie to generate struggling search tasks online. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of ‘repeated similar queries’ and ‘quick-back clicks’, etc. – tasks of diverse topics, high quality and difficulty can be created using this framework. For the benefit of the community, we publicly released the platform, a task set containing 80 topically diverse struggling search tasks generated and examined in this work, and the corresponding anonymized user behavior logs.

Original language	English
Title of host publication	Web Information Systems Engineering
Subtitle of host publication	WISE 2020 - Proceedings
Editors	Zhisheng Huang, Wouter Beek, Hua Wang, Yanchun Zhang, Rui Zhou
Place of Publication	Cham
Publisher	Springer
Pages	3-20
Number of pages	18
ISBN (Electronic)	978-3-030-62008-0
ISBN (Print)	978-3-030-62007-3
DOIs	https://doi.org/10.1007/978-3-030-62008-0_1
Publication status	Published - 2020
Event	21st International Conference on Web Information Systems Engineering, WISE 2020 - Amsterdam, Netherlands Duration: 20 Oct 2020 → 24 Oct 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Publisher	Springer
Volume	12343
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	21st International Conference on Web Information Systems Engineering, WISE 2020
Country/Territory	Netherlands
City	Amsterdam
Period	20/10/20 → 24/10/20

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

SERPs
User behavior
User interaction
Web search

Access to Document

10.1007/978-3-030-62008-0_1

Xu2020_Chapter_TaskGenieCrowd_PoweredTaskGeneFinal published version, 950 KB

Cite this

Xu, L., Zhou, X., & Gadiraju, U. (2020). TaskGenie: Crowd-Powered Task Generation for Struggling Search. In Z. Huang, W. Beek, H. Wang, Y. Zhang, & R. Zhou (Eds.), Web Information Systems Engineering: WISE 2020 - Proceedings (pp. 3-20). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12343 ). Springer. https://doi.org/10.1007/978-3-030-62008-0_1

Xu, Luyan ; Zhou, Xuan ; Gadiraju, Ujwal. / TaskGenie : Crowd-Powered Task Generation for Struggling Search. Web Information Systems Engineering: WISE 2020 - Proceedings. editor / Zhisheng Huang ; Wouter Beek ; Hua Wang ; Yanchun Zhang ; Rui Zhou. Cham : Springer, 2020. pp. 3-20 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{90bd425f220a42d2ba064c6091b77db1,

title = "TaskGenie: Crowd-Powered Task Generation for Struggling Search",

abstract = "Search tasks provide a medium for the evaluation of system performance and the underlying analytical aspects of IR systems. Researchers have recently developed new interfaces or mechanisms to support vague information needs and struggling search. However, little attention has been paid to the generation of a unified task set for evaluation and comparison of search engine improvements for struggling search. Generation of such tasks is inherently difficult, as each task is supposed to trigger struggling and exploring user behavior rather than simple search behavior. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for system evaluation. In this paper, we propose a task generation method and develop a crowd-powered platform called TaskGenie to generate struggling search tasks online. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of {\textquoteleft}repeated similar queries{\textquoteright} and {\textquoteleft}quick-back clicks{\textquoteright}, etc. – tasks of diverse topics, high quality and difficulty can be created using this framework. For the benefit of the community, we publicly released the platform, a task set containing 80 topically diverse struggling search tasks generated and examined in this work, and the corresponding anonymized user behavior logs.",

keywords = "SERPs, User behavior, User interaction, Web search",

author = "Luyan Xu and Xuan Zhou and Ujwal Gadiraju",

note = "Green Open Access added to TU Delft Institutional Repository {\textquoteleft}You share, we take care!{\textquoteright} – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. ; 21st International Conference on Web Information Systems Engineering, WISE 2020 ; Conference date: 20-10-2020 Through 24-10-2020",

year = "2020",

doi = "10.1007/978-3-030-62008-0_1",

language = "English",

isbn = "978-3-030-62007-3",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "3--20",

editor = "Zhisheng Huang and Wouter Beek and Hua Wang and Yanchun Zhang and Rui Zhou",

booktitle = "Web Information Systems Engineering",

}

Xu, L, Zhou, X & Gadiraju, U 2020, TaskGenie: Crowd-Powered Task Generation for Struggling Search. in Z Huang, W Beek, H Wang, Y Zhang & R Zhou (eds), Web Information Systems Engineering: WISE 2020 - Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12343 , Springer, Cham, pp. 3-20, 21st International Conference on Web Information Systems Engineering, WISE 2020, Amsterdam, Netherlands, 20/10/20. https://doi.org/10.1007/978-3-030-62008-0_1

TaskGenie: Crowd-Powered Task Generation for Struggling Search. / Xu, Luyan; Zhou, Xuan; Gadiraju, Ujwal.
Web Information Systems Engineering: WISE 2020 - Proceedings. ed. / Zhisheng Huang; Wouter Beek; Hua Wang; Yanchun Zhang; Rui Zhou. Cham: Springer, 2020. p. 3-20 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12343 ).

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - TaskGenie

T2 - 21st International Conference on Web Information Systems Engineering, WISE 2020

AU - Xu, Luyan

AU - Zhou, Xuan

AU - Gadiraju, Ujwal

N1 - Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2020

Y1 - 2020

N2 - Search tasks provide a medium for the evaluation of system performance and the underlying analytical aspects of IR systems. Researchers have recently developed new interfaces or mechanisms to support vague information needs and struggling search. However, little attention has been paid to the generation of a unified task set for evaluation and comparison of search engine improvements for struggling search. Generation of such tasks is inherently difficult, as each task is supposed to trigger struggling and exploring user behavior rather than simple search behavior. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for system evaluation. In this paper, we propose a task generation method and develop a crowd-powered platform called TaskGenie to generate struggling search tasks online. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of ‘repeated similar queries’ and ‘quick-back clicks’, etc. – tasks of diverse topics, high quality and difficulty can be created using this framework. For the benefit of the community, we publicly released the platform, a task set containing 80 topically diverse struggling search tasks generated and examined in this work, and the corresponding anonymized user behavior logs.

AB - Search tasks provide a medium for the evaluation of system performance and the underlying analytical aspects of IR systems. Researchers have recently developed new interfaces or mechanisms to support vague information needs and struggling search. However, little attention has been paid to the generation of a unified task set for evaluation and comparison of search engine improvements for struggling search. Generation of such tasks is inherently difficult, as each task is supposed to trigger struggling and exploring user behavior rather than simple search behavior. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for system evaluation. In this paper, we propose a task generation method and develop a crowd-powered platform called TaskGenie to generate struggling search tasks online. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of ‘repeated similar queries’ and ‘quick-back clicks’, etc. – tasks of diverse topics, high quality and difficulty can be created using this framework. For the benefit of the community, we publicly released the platform, a task set containing 80 topically diverse struggling search tasks generated and examined in this work, and the corresponding anonymized user behavior logs.

KW - SERPs

KW - User behavior

KW - User interaction

KW - Web search

UR - http://www.scopus.com/inward/record.url?scp=85096531904&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-62008-0_1

DO - 10.1007/978-3-030-62008-0_1

M3 - Conference contribution

AN - SCOPUS:85096531904

SN - 978-3-030-62007-3

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 3

EP - 20

BT - Web Information Systems Engineering

A2 - Huang, Zhisheng

A2 - Beek, Wouter

A2 - Wang, Hua

A2 - Zhang, Yanchun

A2 - Zhou, Rui

PB - Springer

CY - Cham

Y2 - 20 October 2020 through 24 October 2020

ER -

Xu L, Zhou X, Gadiraju U. TaskGenie: Crowd-Powered Task Generation for Struggling Search. In Huang Z, Beek W, Wang H, Zhang Y, Zhou R, editors, Web Information Systems Engineering: WISE 2020 - Proceedings. Cham: Springer. 2020. p. 3-20. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-62008-0_1

TaskGenie: Crowd-Powered Task Generation for Struggling Search

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this