TaskGenie: Crowd-Powered Task Generation for Struggling Search

Luyan Xu, Xuan Zhou, Ujwal Gadiraju

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

Abstract

Search tasks provide a medium for the evaluation of system performance and the underlying analytical aspects of IR systems. Researchers have recently developed new interfaces or mechanisms to support vague information needs and struggling search. However, little attention has been paid to the generation of a unified task set for evaluation and comparison of search engine improvements for struggling search. Generation of such tasks is inherently difficult, as each task is supposed to trigger struggling and exploring user behavior rather than simple search behavior. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for system evaluation. In this paper, we propose a task generation method and develop a crowd-powered platform called TaskGenie to generate struggling search tasks online. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of ‘repeated similar queries’ and ‘quick-back clicks’, etc. – tasks of diverse topics, high quality and difficulty can be created using this framework. For the benefit of the community, we publicly released the platform, a task set containing 80 topically diverse struggling search tasks generated and examined in this work, and the corresponding anonymized user behavior logs.

Original languageEnglish
Title of host publicationWeb Information Systems Engineering
Subtitle of host publicationWISE 2020 - Proceedings
EditorsZhisheng Huang, Wouter Beek, Hua Wang, Yanchun Zhang, Rui Zhou
Place of PublicationCham
PublisherSpringer
Pages3-20
Number of pages18
ISBN (Electronic)978-3-030-62008-0
ISBN (Print)978-3-030-62007-3
DOIs
Publication statusPublished - 2020
Event21st International Conference on Web Information Systems Engineering, WISE 2020 - Amsterdam, Netherlands
Duration: 20 Oct 202024 Oct 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PublisherSpringer
Volume12343
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference21st International Conference on Web Information Systems Engineering, WISE 2020
CountryNetherlands
CityAmsterdam
Period20/10/2024/10/20

Keywords

  • SERPs
  • User behavior
  • User interaction
  • Web search

Fingerprint Dive into the research topics of 'TaskGenie: Crowd-Powered Task Generation for Struggling Search'. Together they form a unique fingerprint.

Cite this