Questionnaire Items for Evaluating Artificial Social Agents - Expert Generated, Content Validated and Reliability Analysed

Siska Fitrianie, Merijn Bruijnes, Fengxiang Li, Willem Paul Brinkman

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

2 Downloads (Pure)

Abstract

In this paper, we report on the multi-year Intelligent Virtual Agents (IVA) community effort, involving more than 90 researchers worldwide, researching the IVA community interests and practice in evaluating human interaction with an artificial social agent (ASA). The joint efforts have previously generated a unified set of 19 constructs that capture more than 80% of constructs used in empirical studies published in the IVA conference between 2013 to 2018. In this paper, we present expert-content-validated 131 questionnaire items for the constructs and their dimensions, and investigate the level of reliability. We establish this in three phases. Firstly, eight experts generated 431 potential construct items. Secondly, 20 experts rated whether items measure (only) their intended construct, resulting in 207 content-validated items. Next, a reliability analysis was conducted, involving 192 crowd-workers who were asked to rate a human interaction with an ASA, which resulted in 131 items (about 5 items per measurement, with Cronbach's alpha ranged [.60 - .87]). These are the starting points for the questionnaire instrument of human-ASA interaction.

Original languageEnglish
Title of host publicationProceedings of the 21st ACM International Conference on Intelligent Virtual Agents, IVA 2021
PublisherAssociation for Computing Machinery (ACM)
Pages84-86
Number of pages3
ISBN (Electronic)9781450386197
DOIs
Publication statusPublished - 2021
Event21st ACM International Conference on Intelligent Virtual Agents, IVA 2021 - Virtual, Online, Japan
Duration: 14 Sep 202117 Sep 2021

Publication series

NameProceedings of the 21st ACM International Conference on Intelligent Virtual Agents, IVA 2021

Conference

Conference21st ACM International Conference on Intelligent Virtual Agents, IVA 2021
Country/TerritoryJapan
CityVirtual, Online
Period14/09/2117/09/21

Keywords

  • Artificial social agent
  • evaluation instrument
  • questionnaire
  • reliability analysis
  • user study

Fingerprint

Dive into the research topics of 'Questionnaire Items for Evaluating Artificial Social Agents - Expert Generated, Content Validated and Reliability Analysed'. Together they form a unique fingerprint.

Cite this