Ready Player One! Eliciting Diverse Knowledge Using A Configurable Game

Agathe Balayn*, Gaole He, Andrea Hu, Jie Yang, Ujwal Gadiraju

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

1 Citation (Scopus)
11 Downloads (Pure)


Access to commonsense knowledge is receiving renewed interest for developing neuro-symbolic AI systems, or debugging deep learning models. Little is currently understood about the types of knowledge that can be gathered using existing knowledge elicitation methods. Moreover, these methods fall short of meeting the evolving requirements of several downstream AI tasks. To this end, collecting broad and tacit knowledge, in addition to negative or discriminative knowledge can be highly useful. Addressing this research gap, we developed a novel game with a purpose, 'FindItOut', to elicit different types of knowledge from human players through easily configurable game mechanics. We recruited 125 players from a crowdsourcing platform, who played 2430 rounds, resulting in the creation of more than 150k tuples of knowledge. Through an extensive evaluation of these tuples, we show that FindItOut can successfully result in the creation of plural knowledge with a good player experience. We evaluate the efficiency of the game (over 10 × higher than a reference baseline) and the usefulness of the resulting knowledge, through the lens of two downstream tasks - commonsense question answering and the identification of discriminative attributes. Finally, we present a rigorous qualitative analysis of the tuples' characteristics, that informs the future use of FindItOut across various researcher and practitioner communities.

Original languageEnglish
Title of host publicationWWW 2022
Subtitle of host publicationProceedings of the ACM Web Conference 2022
EditorsF. Laforest, R. Troncy
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
Number of pages11
ISBN (Print)978-1-4503-9096-5
Publication statusPublished - 2022
Event31st ACM World Wide Web Conference, WWW 2022 - Virtual, Online at Lyon, France
Duration: 25 Apr 202229 Apr 2022


Conference31st ACM World Wide Web Conference, WWW 2022
CityVirtual, Online at Lyon


  • commonsense
  • discriminative knowledge
  • GWAP
  • human computation
  • knowledge elicitation
  • neuro-symbolic AI


Dive into the research topics of 'Ready Player One! Eliciting Diverse Knowledge Using A Configurable Game'. Together they form a unique fingerprint.

Cite this