Data and Analysis Underlying the Research into Establishing Reference Points for Artificial Social Agent Evaluation: The ASAQ Representative Set 2025

We present data, scripts, and analysis based on a study into the ASAQ concurrent validity and normative dataset. They are complementary materials of a publication:

Fitrianie S, Abdulrahman A, Bruijnes M and Brinkman W-P (2025) Establishing reference points for artificial social agent evaluation: the ASAQ representative set 2025.

Front. Comput. Sci. 7:1726078. doi: 10.3389/fcomp.2025.1726078

The study was structured into two distinct yet interconnected phases designed to systematically investigate public familiarity and experiences with contemporary Artificial Social Agents (ASAs). In the initial screening phase, a diverse participant pool was surveyed regarding their prior interactions with 14 different ASAs currently available in the market. The subsequent and more comprehensive evaluation phase focused on ASAQ concurrent validity and establishing the ASAQ Representative Set 2025. During this phase, qualified participants from the initial screening were asked to provide detailed assessments of their experiences with one specific ASA they had previously encountered. Each participant completed two key evaluation components: first, they responded to the newly developed Artificial Social Agent Questionnaire (ASAQ), and second, they completed one assigned questionnaire from a set of eight established questionnaires.

The study protocol received full ethical approval from the Human Research Ethics Committee at Delft University of Technology (Approval No. 2685, dated January 13, 2023). For a complete description of the experimental design, please refer to:

S. Fitrianie, M. Bruijnes, A. Abdulrahman, F. Li, and W.-P. Brinkman. Study 9: Concurrent validation and a normative dataset development. OSF Registries https://archive.org/details/osf-registrations-6gz29-v1, DOI: https://doi.org/10.17605/OSF.IO/6GZ29, 2024. Open-Ended Registration.


Parts of this work were funded by the 4TU Pride and Prejudice project. We acknowledge the efforts of all OSF workgroup members, and in particular mention the large recurring efforts of Nele Albers, Andrea Bönsch, Jonathan Ehret, Fengxian Li, and Deborah Richards.

Abstract of Fitrianie et al., 2025:

To support the development of Artificial Social Agents (ASAs), we created the “ASAQ Representative Set 2025,” a benchmark dataset capturing user experiences with widely known agents. We first surveyed 1,253 individuals to identify the most commonly used agents from an initial list of 14. From this, we selected nine agents and a dog for comparison. Then, 666 participants reported on their interactions with these agents using the ASA Questionnaire. This dataset offers a reference for understanding how new ASAs compare to familiar ones like ChatGPT, Siri, and Roomba. We present both the survey results and the benchmark data set.