Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment

Merle M. Reimann; Florian A. Kunneman; Catharine Oertel; Koen V. Hindriks

doi:10.1145/3719160.3736629

Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment

Merle M. Reimann^*, Florian A. Kunneman, Catharine Oertel, Koen V. Hindriks

^*Corresponding author for this work

Interactive Intelligence

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

3 Downloads (Pure)

Abstract

When a user interacts with a conversational agent for the first time, they may not be aware of the agent’s capabilities, leading to suboptimal use or interaction breakdowns. To avoid a mismatch with the actual capabilities, the agent’s capabilities have to be made transparent to the user. To investigate whether communication of an agent’s capabilities during interactions enhances transparency and improves the user’s mental model, we conducted a user study with 56 participants. Each participant had three speech-based interactions with an agent that communicated its capabilities or an agent that did not. Our results suggest that the communication led to a change in user behavior with significantly longer utterances. However, the users’ mental models of the agent’s capabilities were not significantly different between the conditions. Participants were able to significantly improve their knowledge of the agent’s capabilities by aligning their mental model over time in both conditions.

Original language	English
Title of host publication	CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces
Editors	J. Sin, E. Law, J. Wallace, C. Munteanu, D. Korre
Place of Publication	New York
Publisher	Association for Computing Machinery, Inc
Pages	1-12
Number of pages	12
ISBN (Print)	979-8-4007-1527-3
DOIs	https://doi.org/10.1145/3719160.3736629
Publication status	Published - 2025
Event	7th Conference on Conversational User Interfaces, CUI 2025 - Waterloo, Canada Duration: 8 Jul 2025 → 10 Jul 2025

Conference

Conference	7th Conference on Conversational User Interfaces, CUI 2025
Country/Territory	Canada
City	Waterloo
Period	8/07/25 → 10/07/25

Keywords

Capability Communication
Conversational Agents
Dialogue Management
Mental Models
Transparency

Access to Document

10.1145/3719160.3736629

3719160.3736629Final published version, 960 KBLicence: CC BY

Cite this

Reimann, M. M., Kunneman, F. A., Oertel, C., & Hindriks, K. V. (2025). Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment. In J. Sin, E. Law, J. Wallace, C. Munteanu, & D. Korre (Eds.), CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces (pp. 1-12). Article 48 Association for Computing Machinery, Inc. https://doi.org/10.1145/3719160.3736629

Reimann, Merle M. ; Kunneman, Florian A. ; Oertel, Catharine et al. / Transparent Conversational Agents : The Impact of Capability Communication on User Behavior and Mental Model Alignment. CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces. editor / J. Sin ; E. Law ; J. Wallace ; C. Munteanu ; D. Korre. New York : Association for Computing Machinery, Inc, 2025. pp. 1-12

@inproceedings{a1909de8dbf846b28ad1e8c4e6545ff7,

title = "Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment",

abstract = "When a user interacts with a conversational agent for the first time, they may not be aware of the agent{\textquoteright}s capabilities, leading to suboptimal use or interaction breakdowns. To avoid a mismatch with the actual capabilities, the agent{\textquoteright}s capabilities have to be made transparent to the user. To investigate whether communication of an agent{\textquoteright}s capabilities during interactions enhances transparency and improves the user{\textquoteright}s mental model, we conducted a user study with 56 participants. Each participant had three speech-based interactions with an agent that communicated its capabilities or an agent that did not. Our results suggest that the communication led to a change in user behavior with significantly longer utterances. However, the users{\textquoteright} mental models of the agent{\textquoteright}s capabilities were not significantly different between the conditions. Participants were able to significantly improve their knowledge of the agent{\textquoteright}s capabilities by aligning their mental model over time in both conditions.",

keywords = "Capability Communication, Conversational Agents, Dialogue Management, Mental Models, Transparency",

author = "Reimann, {Merle M.} and Kunneman, {Florian A.} and Catharine Oertel and Hindriks, {Koen V.}",

year = "2025",

doi = "10.1145/3719160.3736629",

language = "English",

isbn = "979-8-4007-1527-3",

pages = "1--12",

editor = "J. Sin and E. Law and J. Wallace and C. Munteanu and D. Korre",

booktitle = "CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces",

publisher = "Association for Computing Machinery, Inc",

note = "7th Conference on Conversational User Interfaces, CUI 2025 ; Conference date: 08-07-2025 Through 10-07-2025",

}

Reimann, MM, Kunneman, FA, Oertel, C & Hindriks, KV 2025, Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment. in J Sin, E Law, J Wallace, C Munteanu & D Korre (eds), CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces., 48, Association for Computing Machinery, Inc, New York, pp. 1-12, 7th Conference on Conversational User Interfaces, CUI 2025, Waterloo, Canada, 8/07/25. https://doi.org/10.1145/3719160.3736629

Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment. / Reimann, Merle M.; Kunneman, Florian A.; Oertel, Catharine et al.
CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces. ed. / J. Sin; E. Law; J. Wallace; C. Munteanu; D. Korre. New York: Association for Computing Machinery, Inc, 2025. p. 1-12 48.

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Transparent Conversational Agents

T2 - 7th Conference on Conversational User Interfaces, CUI 2025

AU - Reimann, Merle M.

AU - Kunneman, Florian A.

AU - Oertel, Catharine

AU - Hindriks, Koen V.

PY - 2025

Y1 - 2025

N2 - When a user interacts with a conversational agent for the first time, they may not be aware of the agent’s capabilities, leading to suboptimal use or interaction breakdowns. To avoid a mismatch with the actual capabilities, the agent’s capabilities have to be made transparent to the user. To investigate whether communication of an agent’s capabilities during interactions enhances transparency and improves the user’s mental model, we conducted a user study with 56 participants. Each participant had three speech-based interactions with an agent that communicated its capabilities or an agent that did not. Our results suggest that the communication led to a change in user behavior with significantly longer utterances. However, the users’ mental models of the agent’s capabilities were not significantly different between the conditions. Participants were able to significantly improve their knowledge of the agent’s capabilities by aligning their mental model over time in both conditions.

AB - When a user interacts with a conversational agent for the first time, they may not be aware of the agent’s capabilities, leading to suboptimal use or interaction breakdowns. To avoid a mismatch with the actual capabilities, the agent’s capabilities have to be made transparent to the user. To investigate whether communication of an agent’s capabilities during interactions enhances transparency and improves the user’s mental model, we conducted a user study with 56 participants. Each participant had three speech-based interactions with an agent that communicated its capabilities or an agent that did not. Our results suggest that the communication led to a change in user behavior with significantly longer utterances. However, the users’ mental models of the agent’s capabilities were not significantly different between the conditions. Participants were able to significantly improve their knowledge of the agent’s capabilities by aligning their mental model over time in both conditions.

KW - Capability Communication

KW - Conversational Agents

KW - Dialogue Management

KW - Mental Models

KW - Transparency

UR - http://www.scopus.com/inward/record.url?scp=105011589173&partnerID=8YFLogxK

U2 - 10.1145/3719160.3736629

DO - 10.1145/3719160.3736629

M3 - Conference contribution

AN - SCOPUS:105011589173

SN - 979-8-4007-1527-3

SP - 1

EP - 12

BT - CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces

A2 - Sin, J.

A2 - Law, E.

A2 - Wallace, J.

A2 - Munteanu, C.

A2 - Korre, D.

PB - Association for Computing Machinery, Inc

CY - New York

Y2 - 8 July 2025 through 10 July 2025

ER -

Reimann MM, Kunneman FA, Oertel C, Hindriks KV. Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment. In Sin J, Law E, Wallace J, Munteanu C, Korre D, editors, CUI 2025 - Proceedings of the 2025 ACM Conference on Conversational User Interfaces. New York: Association for Computing Machinery, Inc. 2025. p. 1-12. 48 doi: 10.1145/3719160.3736629

Transparent Conversational Agents: The Impact of Capability Communication on User Behavior and Mental Model Alignment

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this