Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms

Mario Scriminaci; Andreas Lommatzsch; Benjamin Kille; Frank Hopfgartner; Martha Larson; Davide Malagoli; András Serény; Till Plumbaum

Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms

Mario Scriminaci, Andreas Lommatzsch, Benjamin Kille, Frank Hopfgartner, Martha Larson, Davide Malagoli, András Serény, Till Plumbaum

Multimedia Computing

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

2 Citations (Scopus)

Abstract

In real-world scenarios, recommenders face non-functional requirementsof technical nature and must handle dynamic data in the formof sequential streams. Evaluation of recommender systems musttake these issues into account in order to be maximally informative.In this paper, we present Idomaar—a framework that enables theefficient multi-dimensional benchmarking of recommender algorithms.Idomaar goes beyond current academic research practicesby creating a realistic evaluation environment and computing botheffectiveness and technical metrics for stream-based as well as setbasedevaluation. A scenario focussing on “research to prototypingto productization” cycle at a company illustrates Idomaar’s potential.We show that Idomaar simplifies testing with varying configurationsand supports flexible integration of different data.

Original language	English
Title of host publication	Poster Proceedings of ACM RecSys 2016
Subtitle of host publication	Proceedings of the Poster Track of the 10th ACM Conference on Recommender Systems, RecSys 2016
Editors	Ido Guy, Amit Sharma
Place of Publication	Aachen
Publisher	CEUR-WS
Pages	1-2
Number of pages	2
Publication status	Published - Sept 2016
Event	10th ACM Conference on Recommender Systems, RecSys 2016 - MIT, Boston, MA, United States Duration: 15 Sept 2016 → 19 Sept 2016 https://recsys.acm.org/recsys16/

Publication series

Name	CEUR Workshop Proceedings
Volume	1688
ISSN (Print)	1613-0073

Conference

Conference	10th ACM Conference on Recommender Systems, RecSys 2016
Country/Territory	United States
City	Boston, MA
Period	15/09/16 → 19/09/16
Internet address	https://recsys.acm.org/recsys16/

Access to Document

http://eprints.gla.ac.uk/121693/

Cite this

Scriminaci, M., Lommatzsch, A., Kille, B., Hopfgartner, F., Larson, M., Malagoli, D., Serény, A., & Plumbaum, T. (2016). Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms. In I. Guy, & A. Sharma (Eds.), Poster Proceedings of ACM RecSys 2016: Proceedings of the Poster Track of the 10th ACM Conference on Recommender Systems, RecSys 2016 (pp. 1-2). (CEUR Workshop Proceedings; Vol. 1688). CEUR-WS. http://eprints.gla.ac.uk/121693/

Scriminaci, Mario ; Lommatzsch, Andreas ; Kille, Benjamin et al. / Idomaar : A Framework for Multi-dimensional Benchmarking of Recommender Algorithms. Poster Proceedings of ACM RecSys 2016: Proceedings of the Poster Track of the 10th ACM Conference on Recommender Systems, RecSys 2016. editor / Ido Guy ; Amit Sharma. Aachen : CEUR-WS, 2016. pp. 1-2 (CEUR Workshop Proceedings).

@inproceedings{cbc9097ed3fd4b80afbf53d449780e9c,

title = "Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms",

abstract = "In real-world scenarios, recommenders face non-functional requirementsof technical nature and must handle dynamic data in the formof sequential streams. Evaluation of recommender systems musttake these issues into account in order to be maximally informative.In this paper, we present Idomaar—a framework that enables theefficient multi-dimensional benchmarking of recommender algorithms.Idomaar goes beyond current academic research practicesby creating a realistic evaluation environment and computing botheffectiveness and technical metrics for stream-based as well as setbasedevaluation. A scenario focussing on “research to prototypingto productization” cycle at a company illustrates Idomaar{\textquoteright}s potential.We show that Idomaar simplifies testing with varying configurationsand supports flexible integration of different data.",

author = "Mario Scriminaci and Andreas Lommatzsch and Benjamin Kille and Frank Hopfgartner and Martha Larson and Davide Malagoli and Andr{\'a}s Ser{\'e}ny and Till Plumbaum",

year = "2016",

month = sep,

language = "English",

series = "CEUR Workshop Proceedings",

publisher = "CEUR-WS",

pages = "1--2",

editor = "Ido Guy and Amit Sharma",

booktitle = "Poster Proceedings of ACM RecSys 2016",

note = "10th ACM Conference on Recommender Systems, RecSys 2016 ; Conference date: 15-09-2016 Through 19-09-2016",

url = "https://recsys.acm.org/recsys16/",

}

Scriminaci, M, Lommatzsch, A, Kille, B, Hopfgartner, F, Larson, M, Malagoli, D, Serény, A & Plumbaum, T 2016, Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms. in I Guy & A Sharma (eds), Poster Proceedings of ACM RecSys 2016: Proceedings of the Poster Track of the 10th ACM Conference on Recommender Systems, RecSys 2016. CEUR Workshop Proceedings, vol. 1688, CEUR-WS, Aachen, pp. 1-2, 10th ACM Conference on Recommender Systems, RecSys 2016, Boston, MA, United States, 15/09/16. <http://eprints.gla.ac.uk/121693/>

Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms. / Scriminaci, Mario; Lommatzsch, Andreas; Kille, Benjamin et al.
Poster Proceedings of ACM RecSys 2016: Proceedings of the Poster Track of the 10th ACM Conference on Recommender Systems, RecSys 2016. ed. / Ido Guy; Amit Sharma. Aachen: CEUR-WS, 2016. p. 1-2 (CEUR Workshop Proceedings; Vol. 1688).

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Idomaar

T2 - 10th ACM Conference on Recommender Systems, RecSys 2016

AU - Scriminaci, Mario

AU - Lommatzsch, Andreas

AU - Kille, Benjamin

AU - Hopfgartner, Frank

AU - Larson, Martha

AU - Malagoli, Davide

AU - Serény, András

AU - Plumbaum, Till

PY - 2016/9

Y1 - 2016/9

N2 - In real-world scenarios, recommenders face non-functional requirementsof technical nature and must handle dynamic data in the formof sequential streams. Evaluation of recommender systems musttake these issues into account in order to be maximally informative.In this paper, we present Idomaar—a framework that enables theefficient multi-dimensional benchmarking of recommender algorithms.Idomaar goes beyond current academic research practicesby creating a realistic evaluation environment and computing botheffectiveness and technical metrics for stream-based as well as setbasedevaluation. A scenario focussing on “research to prototypingto productization” cycle at a company illustrates Idomaar’s potential.We show that Idomaar simplifies testing with varying configurationsand supports flexible integration of different data.

AB - In real-world scenarios, recommenders face non-functional requirementsof technical nature and must handle dynamic data in the formof sequential streams. Evaluation of recommender systems musttake these issues into account in order to be maximally informative.In this paper, we present Idomaar—a framework that enables theefficient multi-dimensional benchmarking of recommender algorithms.Idomaar goes beyond current academic research practicesby creating a realistic evaluation environment and computing botheffectiveness and technical metrics for stream-based as well as setbasedevaluation. A scenario focussing on “research to prototypingto productization” cycle at a company illustrates Idomaar’s potential.We show that Idomaar simplifies testing with varying configurationsand supports flexible integration of different data.

M3 - Conference contribution

T3 - CEUR Workshop Proceedings

SP - 1

EP - 2

BT - Poster Proceedings of ACM RecSys 2016

A2 - Guy, Ido

A2 - Sharma, Amit

PB - CEUR-WS

CY - Aachen

Y2 - 15 September 2016 through 19 September 2016

ER -

Scriminaci M, Lommatzsch A, Kille B, Hopfgartner F, Larson M, Malagoli D et al. Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms. In Guy I, Sharma A, editors, Poster Proceedings of ACM RecSys 2016: Proceedings of the Poster Track of the 10th ACM Conference on Recommender Systems, RecSys 2016. Aachen: CEUR-WS. 2016. p. 1-2. (CEUR Workshop Proceedings).

Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this