How Stated Accuracy of an AI System and Analogies to Explain Accuracy Affect Human Reliance on the System

Gaole He*, Stefan Buijsman*, Ujwal Gadiraju

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

56 Downloads (Pure)

Abstract

AI systems are increasingly being used to support human decision making. It is important that AI advice is followed appropriately. However, according to existing literature, users typically under-rely or over-rely on AI systems, and this leads to sub-optimal team performance. In this context, we investigate the role of stated system accuracy by contrasting the lack of system information with the presence of system accuracy in a loan prediction task. We explore how the degree to which humans understand system accuracy influences their reliance on the AI system, by investigating numeracy levels and with the aid of analogies to explain system accuracy in a first of its kind between-subjects study (N=281). We found that explaining the stated accuracy of a system using analogies failed to help users rely on the AI systemappropriately (i.e., the tendency of users to rely on the system when the system is correct, or on themselves otherwise). To eliminate the impact of subjective attitudes towards analogy domains, we conducted a within-subjects study (N=248) where each participant worked on tasks with analogy-based explanations from different domains. Results from this second study confirmed that explaining stated accuracy of the system with analogies was not sufficient to facilitate appropriate reliance on the AI system in the context of loan prediction tasks, irrespective of individual user differences. Based on our findings from the two studies, we reason that the under-reliance on the AI system may be a result of users' overestimation of their own ability to solve the given task. Thus, although familiar analogies can be effective in improving the intelligibility of stated accuracy of the system, an improved understanding of system accuracy does not necessarily lead to improved system reliance and team performance.

Original languageEnglish
Article number3610067
JournalProceedings of the ACM on Human-Computer Interaction
Volume7
Issue numberCSCW2
DOIs
Publication statusPublished - 2023

Keywords

  • analogy
  • human-subjects experiment
  • reliance
  • trust

Fingerprint

Dive into the research topics of 'How Stated Accuracy of an AI System and Analogies to Explain Accuracy Affect Human Reliance on the System'. Together they form a unique fingerprint.

Cite this