Abstract
We present the ASA Questionnaire, an instrument for evaluating human interaction with an artificial social agent (ASA), resulting from multi-year efforts involving more than 100 Intelligent Virtual Agent (IVA) researchers worldwide. It has 19 measurement constructs constituted by 90 items, which capture more than 80% of the constructs identified in empirical studies published in the IVA conference 2013 - 2018. This paper reports on construct validity analysis, specifically convergent and discriminant validity of initial 131 instrument items that involved 532 crowd-workers who were asked to rate human interaction with 14 different ASAs. The analysis included several factor analysis models and resulted in the selection of 90 items for inclusion in the long version of the ASA questionnaire. In addition, a representative item of each construct or dimension was selected to create a 24-item short version of the ASA questionnaire. Whereas the long version is suitable for a comprehensive evaluation of human-ASA interaction, the short version allows quick analysis and description of the interaction with the ASA. To support reporting ASA questionnaire results, we also put forward an ASA chart. The chart provides a quick overview of the agent profile.
Original language | English |
---|---|
Title of host publication | IVA 2022 - Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents |
Publisher | Association for Computing Machinery (ACM) |
ISBN (Electronic) | 9781450392488 |
DOIs | |
Publication status | Published - 2022 |
Event | 22nd ACM International Conference on Intelligent Virtual Agents, IVA 2022 - Faro, Portugal Duration: 6 Sept 2022 → 9 Sept 2022 |
Publication series
Name | IVA 2022 - Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents |
---|
Conference
Conference | 22nd ACM International Conference on Intelligent Virtual Agents, IVA 2022 |
---|---|
Country/Territory | Portugal |
City | Faro |
Period | 6/09/22 → 9/09/22 |
Keywords
- artificial social agent
- construct validity
- evaluation instrument
- questionnaire
- user study
Fingerprint
Dive into the research topics of 'The artificial-social-agent questionnaire: Establishing the long and short questionnaire versions'. Together they form a unique fingerprint.Datasets
-
Data and analysis underlying the research into the Artificial-Social-Agent Questionnaire: Establishing the long and short questionnaire versions
Fitrianie, S. (Creator), Brinkman, W. P. (Creator), Bruijnes, M. (Creator), Li, F. (Creator) & Abdulrahman, A. (Creator), TU Delft - 4TU.ResearchData, 22 Jul 2022
DOI: 10.4121/19758436
Dataset/Software: Dataset
-
Artificial Social Agent Questionnaire Instrument
Fitrianie, S. (Creator), Brinkman, W. P. (Creator), Bruijnes, M. (Creator), Li, F. (Creator) & Abdulrahman, A. (Creator), TU Delft - 4TU.ResearchData, 22 Jul 2022
DOI: 10.4121/19650846
Dataset/Software: Dataset