Metrics for Evaluating Explainable Recommender Systems

Joris Hulstijn; Igor Tchappi; Amro Najjar; Reyhan Aydoğan

doi:10.1007/978-3-031-40878-6_12

Metrics for Evaluating Explainable Recommender Systems

Joris Hulstijn^*, Igor Tchappi, Amro Najjar, Reyhan Aydoğan

^*Corresponding author for this work

Interactive Intelligence

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

5 Downloads (Pure)

Abstract

Recommender systems aim to support their users by reducing information overload so that they can make better decisions. Recommender systems must be transparent, so users can form mental models about the system’s goals, internal state, and capabilities, that are in line with their actual design. Explanations and transparent behaviour of the system should inspire trust and, ultimately, lead to more persuasive recommendations. Here, explanations convey reasons why a recommendation is given or how the system forms its recommendations. This paper focuses on the question how such claims about effectiveness of explanations can be evaluated. Accordingly, we investigate various models that are used to assess the effects of explanations and recommendations. We discuss objective and subjective measurement and argue that both are needed. We define a set of metrics for measuring the effectiveness of explanations and recommendations. The feasibility of using these metrics is discussed in the context of a specific explainable recommender system in the food and health domain.

Original language	English
Title of host publication	Explainable and Transparent AI and Multi-Agent Systems - 5th International Workshop, EXTRAAMAS 2023, Revised Selected Papers
Editors	Davide Calvaresi, Amro Najjar, Andrea Omicini, Rachele Carli, Giovanni Ciatto, Reyhan Aydogan, Yazan Mualla, Kary Främling
Publisher	Springer
Pages	212-230
Number of pages	19
ISBN (Print)	9783031408779
DOIs	https://doi.org/10.1007/978-3-031-40878-6_12
Publication status	Published - 2023
Event	Proceedings of the 5th International Workshop on EXTRAAMAS 2023 - London, United Kingdom Duration: 29 May 2023 → 29 May 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14127 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	Proceedings of the 5th International Workshop on EXTRAAMAS 2023
Country/Territory	United Kingdom
City	London
Period	29/05/23 → 29/05/23

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

Evaluation
Explainable AI
Metrics
Recommender systems

Access to Document

10.1007/978-3-031-40878-6_12

978-3-031-40878-6_12Final published version, 538 KB

Cite this

Hulstijn, J., Tchappi, I., Najjar, A., & Aydoğan, R. (2023). Metrics for Evaluating Explainable Recommender Systems. In D. Calvaresi, A. Najjar, A. Omicini, R. Carli, G. Ciatto, R. Aydogan, Y. Mualla, & K. Främling (Eds.), Explainable and Transparent AI and Multi-Agent Systems - 5th International Workshop, EXTRAAMAS 2023, Revised Selected Papers (pp. 212-230). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14127 LNAI). Springer. https://doi.org/10.1007/978-3-031-40878-6_12

Hulstijn, Joris ; Tchappi, Igor ; Najjar, Amro et al. / Metrics for Evaluating Explainable Recommender Systems. Explainable and Transparent AI and Multi-Agent Systems - 5th International Workshop, EXTRAAMAS 2023, Revised Selected Papers. editor / Davide Calvaresi ; Amro Najjar ; Andrea Omicini ; Rachele Carli ; Giovanni Ciatto ; Reyhan Aydogan ; Yazan Mualla ; Kary Främling. Springer, 2023. pp. 212-230 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{4e0e3a28e3694968a01882733ba9b363,

title = "Metrics for Evaluating Explainable Recommender Systems",

abstract = "Recommender systems aim to support their users by reducing information overload so that they can make better decisions. Recommender systems must be transparent, so users can form mental models about the system{\textquoteright}s goals, internal state, and capabilities, that are in line with their actual design. Explanations and transparent behaviour of the system should inspire trust and, ultimately, lead to more persuasive recommendations. Here, explanations convey reasons why a recommendation is given or how the system forms its recommendations. This paper focuses on the question how such claims about effectiveness of explanations can be evaluated. Accordingly, we investigate various models that are used to assess the effects of explanations and recommendations. We discuss objective and subjective measurement and argue that both are needed. We define a set of metrics for measuring the effectiveness of explanations and recommendations. The feasibility of using these metrics is discussed in the context of a specific explainable recommender system in the food and health domain.",

keywords = "Evaluation, Explainable AI, Metrics, Recommender systems",

author = "Joris Hulstijn and Igor Tchappi and Amro Najjar and Reyhan Aydoğan",

note = "Green Open Access added to TU Delft Institutional Repository {\textquoteleft}You share, we take care!{\textquoteright} – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. ; Proceedings of the 5th International Workshop on EXTRAAMAS 2023 ; Conference date: 29-05-2023 Through 29-05-2023",

year = "2023",

doi = "10.1007/978-3-031-40878-6_12",

language = "English",

isbn = "9783031408779",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "212--230",

editor = "Davide Calvaresi and Amro Najjar and Andrea Omicini and Rachele Carli and Giovanni Ciatto and Reyhan Aydogan and Yazan Mualla and Kary Fr{\"a}mling",

booktitle = "Explainable and Transparent AI and Multi-Agent Systems - 5th International Workshop, EXTRAAMAS 2023, Revised Selected Papers",

}

Hulstijn, J, Tchappi, I, Najjar, A & Aydoğan, R 2023, Metrics for Evaluating Explainable Recommender Systems. in D Calvaresi, A Najjar, A Omicini, R Carli, G Ciatto, R Aydogan, Y Mualla & K Främling (eds), Explainable and Transparent AI and Multi-Agent Systems - 5th International Workshop, EXTRAAMAS 2023, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14127 LNAI, Springer, pp. 212-230, Proceedings of the 5th International Workshop on EXTRAAMAS 2023, London, United Kingdom, 29/05/23. https://doi.org/10.1007/978-3-031-40878-6_12

Metrics for Evaluating Explainable Recommender Systems. / Hulstijn, Joris; Tchappi, Igor; Najjar, Amro et al.
Explainable and Transparent AI and Multi-Agent Systems - 5th International Workshop, EXTRAAMAS 2023, Revised Selected Papers. ed. / Davide Calvaresi; Amro Najjar; Andrea Omicini; Rachele Carli; Giovanni Ciatto; Reyhan Aydogan; Yazan Mualla; Kary Främling. Springer, 2023. p. 212-230 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14127 LNAI).

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Metrics for Evaluating Explainable Recommender Systems

AU - Hulstijn, Joris

AU - Tchappi, Igor

AU - Najjar, Amro

AU - Aydoğan, Reyhan

N1 - Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2023

Y1 - 2023

N2 - Recommender systems aim to support their users by reducing information overload so that they can make better decisions. Recommender systems must be transparent, so users can form mental models about the system’s goals, internal state, and capabilities, that are in line with their actual design. Explanations and transparent behaviour of the system should inspire trust and, ultimately, lead to more persuasive recommendations. Here, explanations convey reasons why a recommendation is given or how the system forms its recommendations. This paper focuses on the question how such claims about effectiveness of explanations can be evaluated. Accordingly, we investigate various models that are used to assess the effects of explanations and recommendations. We discuss objective and subjective measurement and argue that both are needed. We define a set of metrics for measuring the effectiveness of explanations and recommendations. The feasibility of using these metrics is discussed in the context of a specific explainable recommender system in the food and health domain.

AB - Recommender systems aim to support their users by reducing information overload so that they can make better decisions. Recommender systems must be transparent, so users can form mental models about the system’s goals, internal state, and capabilities, that are in line with their actual design. Explanations and transparent behaviour of the system should inspire trust and, ultimately, lead to more persuasive recommendations. Here, explanations convey reasons why a recommendation is given or how the system forms its recommendations. This paper focuses on the question how such claims about effectiveness of explanations can be evaluated. Accordingly, we investigate various models that are used to assess the effects of explanations and recommendations. We discuss objective and subjective measurement and argue that both are needed. We define a set of metrics for measuring the effectiveness of explanations and recommendations. The feasibility of using these metrics is discussed in the context of a specific explainable recommender system in the food and health domain.

KW - Evaluation

KW - Explainable AI

KW - Metrics

KW - Recommender systems

UR - http://www.scopus.com/inward/record.url?scp=85172263345&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-40878-6_12

DO - 10.1007/978-3-031-40878-6_12

M3 - Conference contribution

AN - SCOPUS:85172263345

SN - 9783031408779

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 212

EP - 230

BT - Explainable and Transparent AI and Multi-Agent Systems - 5th International Workshop, EXTRAAMAS 2023, Revised Selected Papers

A2 - Calvaresi, Davide

A2 - Najjar, Amro

A2 - Omicini, Andrea

A2 - Carli, Rachele

A2 - Ciatto, Giovanni

A2 - Aydogan, Reyhan

A2 - Mualla, Yazan

A2 - Främling, Kary

PB - Springer

T2 - Proceedings of the 5th International Workshop on EXTRAAMAS 2023

Y2 - 29 May 2023 through 29 May 2023

ER -

Hulstijn J, Tchappi I, Najjar A, Aydoğan R. Metrics for Evaluating Explainable Recommender Systems. In Calvaresi D, Najjar A, Omicini A, Carli R, Ciatto G, Aydogan R, Mualla Y, Främling K, editors, Explainable and Transparent AI and Multi-Agent Systems - 5th International Workshop, EXTRAAMAS 2023, Revised Selected Papers. Springer. 2023. p. 212-230. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-40878-6_12