What Should You Know? A Human-In-the-Loop Approach to Unknown Unknowns Characterization in Image Recognition

Shahin Sharifi Noorian, Sihang Qiu, Ujwal Gadiraju, Jie Yang*, Alessandro Bozzon

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

6 Citations (Scopus)
41 Downloads (Pure)

Abstract

Unknown unknowns represent a major challenge in reliable image recognition. Existing methods mainly focus on unknown unknowns identification, leveraging human intelligence to gather images that are potentially difficult for the machine. To drive a deeper understanding of unknown unknowns and more effective identification and treatment, this paper focuses on unknown unknowns characterization. We introduce a human-in-the-loop, semantic analysis framework for characterizing unknown unknowns at scale. We engage humans in two tasks that specify what a machine should know and describe what it really knows, respectively, both at the conceptual level, supported by information extraction and machine learning interpretability methods. Data partitioning and sampling techniques are employed to scale out human contributions in handling large data. Through extensive experimentation on scene recognition tasks, we show that our approach provides a rich, descriptive characterization of unknown unknowns and allows for more effective and cost-efficient detection than the state of the art.

Original languageEnglish
Title of host publicationWWW 2022 - Proceedings of the ACM Web Conference 2022
PublisherAssociation for Computing Machinery (ACM)
Pages882-892
Number of pages11
ISBN (Electronic)978-1-4503-9096-5
DOIs
Publication statusPublished - 2022
Event31st ACM World Wide Web Conference, WWW 2022 - Virtual, Online at Lyon, France
Duration: 25 Apr 202229 Apr 2022

Publication series

NameWWW 2022 - Proceedings of the ACM Web Conference 2022

Conference

Conference31st ACM World Wide Web Conference, WWW 2022
Country/TerritoryFrance
CityVirtual, Online at Lyon
Period25/04/2229/04/22

Keywords

  • humans in the loop
  • semantic analysis
  • Unknown unknowns

Fingerprint

Dive into the research topics of 'What Should You Know? A Human-In-the-Loop Approach to Unknown Unknowns Characterization in Image Recognition'. Together they form a unique fingerprint.

Cite this