A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms

Benjamin Kille; Andreas Lommatzsch; Frank Hopfgartner; Martha Larson; Arjen P. de Vries

doi:10.1145/3077136.3080726

A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms

Benjamin Kille, Andreas Lommatzsch, Frank Hopfgartner, Martha Larson, Arjen P. de Vries

Multimedia Computing

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

4 Citations (Scopus)

70 Downloads (Pure)

Abstract

Recommender System research has evolved to focus on developing algorithms capable of high performance in online systems. This development calls for a new evaluation infrastructure that supports multi-dimensional evaluation of recommender systems. Today’s researchers should analyze algorithms with respect to a variety of aspects including predictive performance and scalability. Researchers need to subject algorithms to realistic conditions in online A/B tests.
We introduce two resources supporting such evaluation methodologies: the new data set of stream recommendation interactions released for CLEF NewsREEL 2017, and the new Open Recommendation Platform (ORP). The data set allows researchers to study a stream recommendation problem closely by “replaying” it locally, and ORP makes it possible to take this evaluation “live” in a living
lab scenario. Specifically, ORP allows researchers to deploy their algorithms in a live stream to carry out A/B tests. To our knowledge, NewsREEL is the first online news recommender system resource to be put at the disposal of the research community. In order to encourage others to develop comparable resources for a wide range of domains, we present a list of practical lessons learned in the
development of the dataset and ORP.

Original language	English
Title of host publication	Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017
Place of Publication	New York, NY
Publisher	Association for Computing Machinery (ACM)
Pages	1257-1260
Number of pages	4
ISBN (Electronic)	978-1-4503-5022-8
DOIs	https://doi.org/10.1145/3077136.3080726
Publication status	Published - 2017
Event	SIGIR 2017: 40th International ACM SIGIR Conference on Research and Development in Information Retrieval - Tokyo, Japan Duration: 7 Aug 2017 → 11 Aug 2017 http://sigir.org/sigir2017/

Conference

Conference	SIGIR 2017
Country/Territory	Japan
City	Tokyo
Period	7/08/17 → 11/08/17
Internet address	http://sigir.org/sigir2017/

Keywords

streams
recommender system
multi--dimensional benchmarking

Access to Document

10.1145/3077136.3080726

35744048Final published version, 869 KBLicence: CC BY-NC-ND

Cite this

Kille, B., Lommatzsch, A., Hopfgartner, F., Larson, M., & de Vries, A. P. (2017). A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017 (pp. 1257-1260). Association for Computing Machinery (ACM). https://doi.org/10.1145/3077136.3080726

@inproceedings{7430fe8b8f6342339189e0a92d91621e,

title = "A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms",

abstract = "Recommender System research has evolved to focus on developing algorithms capable of high performance in online systems. This development calls for a new evaluation infrastructure that supports multi-dimensional evaluation of recommender systems. Today{\textquoteright}s researchers should analyze algorithms with respect to a variety of aspects including predictive performance and scalability. Researchers need to subject algorithms to realistic conditions in online A/B tests.We introduce two resources supporting such evaluation methodologies: the new data set of stream recommendation interactions released for CLEF NewsREEL 2017, and the new Open Recommendation Platform (ORP). The data set allows researchers to study a stream recommendation problem closely by “replaying” it locally, and ORP makes it possible to take this evaluation “live” in a livinglab scenario. Specifically, ORP allows researchers to deploy their algorithms in a live stream to carry out A/B tests. To our knowledge, NewsREEL is the first online news recommender system resource to be put at the disposal of the research community. In order to encourage others to develop comparable resources for a wide range of domains, we present a list of practical lessons learned in thedevelopment of the dataset and ORP.",

keywords = "streams, recommender system, multi--dimensional benchmarking",

author = "Benjamin Kille and Andreas Lommatzsch and Frank Hopfgartner and Martha Larson and {de Vries}, {Arjen P.}",

year = "2017",

doi = "10.1145/3077136.3080726",

language = "English",

pages = "1257--1260",

booktitle = "Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017",

publisher = "Association for Computing Machinery (ACM)",

address = "United States",

note = "SIGIR 2017 : 40th International ACM SIGIR Conference on Research and Development in Information Retrieval ; Conference date: 07-08-2017 Through 11-08-2017",

url = "http://sigir.org/sigir2017/",

}

Kille, B, Lommatzsch, A, Hopfgartner, F, Larson, M & de Vries, AP 2017, A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms. in Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017. Association for Computing Machinery (ACM), New York, NY, pp. 1257-1260, SIGIR 2017, Tokyo, Japan, 7/08/17. https://doi.org/10.1145/3077136.3080726

A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms. / Kille, Benjamin; Lommatzsch, Andreas; Hopfgartner, Frank et al.
Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017. New York, NY: Association for Computing Machinery (ACM), 2017. p. 1257-1260.

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms

AU - Kille, Benjamin

AU - Lommatzsch, Andreas

AU - Hopfgartner, Frank

AU - Larson, Martha

AU - de Vries, Arjen P.

PY - 2017

Y1 - 2017

N2 - Recommender System research has evolved to focus on developing algorithms capable of high performance in online systems. This development calls for a new evaluation infrastructure that supports multi-dimensional evaluation of recommender systems. Today’s researchers should analyze algorithms with respect to a variety of aspects including predictive performance and scalability. Researchers need to subject algorithms to realistic conditions in online A/B tests.We introduce two resources supporting such evaluation methodologies: the new data set of stream recommendation interactions released for CLEF NewsREEL 2017, and the new Open Recommendation Platform (ORP). The data set allows researchers to study a stream recommendation problem closely by “replaying” it locally, and ORP makes it possible to take this evaluation “live” in a livinglab scenario. Specifically, ORP allows researchers to deploy their algorithms in a live stream to carry out A/B tests. To our knowledge, NewsREEL is the first online news recommender system resource to be put at the disposal of the research community. In order to encourage others to develop comparable resources for a wide range of domains, we present a list of practical lessons learned in thedevelopment of the dataset and ORP.

AB - Recommender System research has evolved to focus on developing algorithms capable of high performance in online systems. This development calls for a new evaluation infrastructure that supports multi-dimensional evaluation of recommender systems. Today’s researchers should analyze algorithms with respect to a variety of aspects including predictive performance and scalability. Researchers need to subject algorithms to realistic conditions in online A/B tests.We introduce two resources supporting such evaluation methodologies: the new data set of stream recommendation interactions released for CLEF NewsREEL 2017, and the new Open Recommendation Platform (ORP). The data set allows researchers to study a stream recommendation problem closely by “replaying” it locally, and ORP makes it possible to take this evaluation “live” in a livinglab scenario. Specifically, ORP allows researchers to deploy their algorithms in a live stream to carry out A/B tests. To our knowledge, NewsREEL is the first online news recommender system resource to be put at the disposal of the research community. In order to encourage others to develop comparable resources for a wide range of domains, we present a list of practical lessons learned in thedevelopment of the dataset and ORP.

KW - streams

KW - recommender system

KW - multi--dimensional benchmarking

UR - http://resolver.tudelft.nl/uuid:7430fe8b-8f63-4233-9189-e0a92d91621e

U2 - 10.1145/3077136.3080726

DO - 10.1145/3077136.3080726

M3 - Conference contribution

SP - 1257

EP - 1260

BT - Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017

PB - Association for Computing Machinery (ACM)

CY - New York, NY

T2 - SIGIR 2017

Y2 - 7 August 2017 through 11 August 2017

ER -

Kille B, Lommatzsch A, Hopfgartner F, Larson M, de Vries AP. A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017. New York, NY: Association for Computing Machinery (ACM). 2017. p. 1257-1260 doi: 10.1145/3077136.3080726

A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this