Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)

Ryan Clancy, Nicola Ferro, Claudia Hauff, Jimmy Lin, Tetsuya Sakai, Ze Zhong Wu

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

4 Citations (Scopus)
6 Downloads (Pure)

Abstract

The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the "jig" that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.

Original languageEnglish
Title of host publicationOSIRRC 2019 The Open-Source IR Replicability Challenge
Subtitle of host publicationProceedings of the Open-Source IR Replicability Challenge (OSIRRC 2019) co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019)
EditorsRyan Clancy, Nicola Ferro, Claudia Hauff, Jimmy Lin, Tetsuya Sakai , Ze Zhong Wu
PublisherCEUR-WS.org
Pages1-7
Number of pages7
Publication statusPublished - 2019
Event2019 Open-Source IR Replicability Challenge, OSIRRC 2019 - Paris, France
Duration: 25 Jul 201925 Jul 2019

Publication series

NameCEUR Workshop Proceedings
Volume2409
ISSN (Print)1613-0073

Conference

Conference2019 Open-Source IR Replicability Challenge, OSIRRC 2019
CountryFrance
CityParis
Period25/07/1925/07/19

Fingerprint Dive into the research topics of 'Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)'. Together they form a unique fingerprint.

  • Cite this

    Clancy, R., Ferro, N., Hauff, C., Lin, J., Sakai, T., & Wu, Z. Z. (2019). Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). In R. Clancy, N. Ferro, C. Hauff, J. Lin, T. Sakai , & Z. Z. Wu (Eds.), OSIRRC 2019 The Open-Source IR Replicability Challenge: Proceedings of the Open-Source IR Replicability Challenge (OSIRRC 2019) co-located with 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019) (pp. 1-7). (CEUR Workshop Proceedings; Vol. 2409). CEUR-WS.org. http://ceur-ws.org/Vol-2409/