Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems

Gaole He, Lucie Kuiper, Ujwal Gadiraju

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

4 Citations (Scopus)
55 Downloads (Pure)

Abstract

The dazzling promises of AI systems to augment humans in various tasks hinge on whether humans can appropriately rely on them. Recent research has shown that appropriate reliance is the key to achieving complementary team performance in AI-assisted decision making. This paper addresses an under-explored problem of whether the Dunning-Kruger Effect (DKE) among people can hinder their appropriate reliance on AI systems. DKE is a metacognitive bias due to which less-competent individuals overestimate their own skill and performance. Through an empirical study (N = 249), we explored the impact of DKE on human reliance on an AI system, and whether such effects can be mitigated using a tutorial intervention that reveals the fallibility of AI advice, and exploiting logic units-based explanations to improve user understanding of AI advice. We found that participants who overestimate their performance tend to exhibit under-reliance on AI systems, which hinders optimal team performance. Logic units-based explanations did not help users in either improving the calibration of their competence or facilitating appropriate reliance. While the tutorial intervention was highly effective in helping users calibrate their self-assessment and facilitating appropriate reliance among participants with overestimated self-assessment, we found that it can potentially hurt the appropriate reliance of participants with underestimated self-assessment. Our work has broad implications on the design of methods to tackle user cognitive biases while facilitating appropriate reliance on AI systems. Our findings advance the current understanding of the role of self-assessment in shaping trust and reliance in human-AI decision making. This lays out promising future directions for relevant HCI research in this community.

Original languageEnglish
Title of host publicationCHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery (ACM)
Number of pages18
ISBN (Electronic)978-1-4503-9421-5
DOIs
Publication statusPublished - 2023
Event2023 CHI Conference on Human Factors in Computing Systems - Congress Center Hamburg (CCH), Hamburg, Germany
Duration: 23 Apr 202328 Apr 2023
https://chi2023.acm.org/

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Conference

Conference2023 CHI Conference on Human Factors in Computing Systems
Abbreviated titleCHI'23
Country/TerritoryGermany
CityHamburg
Period23/04/2328/04/23
Internet address

Keywords

  • Appropriate Reliance
  • Dunning-Kruger Effect
  • Human-AI Decision Making
  • XAI

Fingerprint

Dive into the research topics of 'Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems'. Together they form a unique fingerprint.

Cite this