To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems

Gaole He, Abri Bharos, Ujwal Gadiraju

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

31 Downloads (Pure)

Abstract

Powerful predictive AI systems have demonstrated great potential in augmenting human decision making. Recent empirical work has argued that the vision for optimal human-AI collaboration requires ‘appropriate reliance’ on AI systems. However, accurately estimating the trustworthiness of AI advice at the instance level is quite challenging, especially in the absence of performance feedback pertaining to the AI system. In practice, the performance disparity of machine learning models on out-of-distribution data makes the dataset-specific performance feedback unreliable in human-AI collaboration. Inspired by existing literature on critical thinking and mindsets, we propose debugging an AI system as an intervention to foster appropriate reliance. This paper explores whether a critical evaluation of AI performance within a debugging setting can better calibrate users’ assessment of an AI system. Through a quantitative empirical study (N = 234), we found that our proposed debugging intervention does not work as expected in facilitating appropriate reliance. Instead, we observe a decrease in reliance on the AI system — potentially resulting from an early exposure to the AI system’s weakness. Our findings have important implications for designing effective interventions to facilitate appropriate reliance and better human-AI collaboration.
Original languageEnglish
Title of host publicationHT 2024
Subtitle of host publicationCreative Intelligence - 35th ACM Conference on Hypertext and Social Media
Place of PublicationNew York, NY
PublisherACM
Pages98-105
Number of pages8
ISBN (Electronic)979-8-4007-0595-3
DOIs
Publication statusPublished - 2024
Event35th ACM Conference on Hypertext and Social Media - Institute of Polish and Classical Philology and Adam Mickiewicz University, Poznań, Poland
Duration: 10 Sept 202413 Sept 2024
https://ht.acm.org/ht2024/

Conference

Conference35th ACM Conference on Hypertext and Social Media
Abbreviated titleHT 2024
Country/TerritoryPoland
CityPoznań
Period10/09/2413/09/24
Internet address

Fingerprint

Dive into the research topics of 'To Err Is AI! Debugging as an Intervention to Facilitate Appropriate Reliance on AI Systems'. Together they form a unique fingerprint.

Cite this