Perspective: Leveraging Human Understanding for Identifying and Characterizing Image Atypicality

Shahin Sharifi Noorian, Sihang Qiu, Burcu Sayin, Agathe Balayn, Ujwal Gadiraju, Jie Yang*, Alessandro Bozzon

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

43 Downloads (Pure)

Abstract

High-quality data plays a vital role in developing reliable image classification models. Despite that, what makes an image difficult to classify remains an unstudied topic. This paper provides a first-of-its-kind, model-agnostic characterization of image atypicality based on human understanding. We consider the setting of image classification "in the wild", where a large number of unlabeled images are accessible, and introduce a scalable and effective human computation approach for proactive identification and characterization of atypical images. Our approach consists of i) an image atypicality identification and characterization task that presents to the human worker both a local view of visually similar images and a global view of images from the class of interest and ii) an automatic image sampling method that selects a diverse set of atypical images based on both visual and semantic features. We demonstrate the effectiveness and cost-efficiency of our approach through controlled crowdsourcing experiments and provide a characterization of image atypicality based on human annotations of 10K images. We showcase the utility of the identified atypical images by testing state-of-the-art image classification services against such images and provide an in-depth comparative analysis of the alignment between human- and machine-perceived image atypicality. Our findings have important implications for developing and deploying reliable image classification systems.

Original languageEnglish
Title of host publicationIUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces
Place of PublicationNew York
PublisherAssociation for Computing Machinery (ACM)
Pages650-663
Number of pages14
ISBN (Print)979-8-4007-0106-1
DOIs
Publication statusPublished - 2023
Event28th International Conference on Intelligent User Interfaces, IUI 2023 - Sydney, Australia
Duration: 27 Mar 202331 Mar 2023

Conference

Conference28th International Conference on Intelligent User Interfaces, IUI 2023
Country/TerritoryAustralia
CitySydney
Period27/03/2331/03/23

Keywords

  • humans in the loop
  • image atypicality
  • Image classification
  • machine learning in the wild

Fingerprint

Dive into the research topics of 'Perspective: Leveraging Human Understanding for Identifying and Characterizing Image Atypicality'. Together they form a unique fingerprint.

Cite this