An in-depth analysis of passage-level label transfer for contextual document ranking

Koustav Rudra*, Zeon Trevor Fernando, Avishek Anand

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Pre-trained contextual language models such as BERT, GPT, and XLnet work quite well for document retrieval tasks. Such models are fine-tuned based on the query-document/query-passage level relevance labels to capture the ranking signals. However, the documents are longer than the passages and such document ranking models suffer from the token limitation (512) of BERT. Researchers proposed ranking strategies that either truncate the documents beyond the token limit or chunk the documents into units that can fit into the BERT. In the later case, the relevance labels are either directly transferred from the original query-document pair or learned through some external model. In this paper, we conduct a detailed study of the design decisions about splitting and label transfer on retrieval effectiveness and efficiency. We find that direct transfer of relevance labels from documents to passages introduces label noise that strongly affects retrieval effectiveness for large training datasets. We also find that query processing times are adversely affected by fine-grained splitting schemes. As a remedy, we propose a careful passage level labelling scheme using weak supervision that delivers improved performance (3–14% in terms of nDCG score) over most of the recently proposed models for ad-hoc retrieval while maintaining manageable computational complexity on four diverse document retrieval datasets.

Original languageEnglish
Article number13
Number of pages24
JournalInformation Retrieval Journal
Volume26
Issue number1-2
DOIs
Publication statusPublished - 2023

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Funding

This paper is an extended work of our previously published paper “Distant Supervision in BERT-based Ad-hoc Document Retrieval”. Koustav Rudra and Avishek Anand. In Proc. CIKM 2020. 2197-2200. In this paper, we extended this work over other datasets to understand the efficiency-efficacy trade-offs of different passage granularity. This work is supported in part by the Science and Engineering Research Board, Department of Science and Technology, Government of India, under Project SRG/2022/001548. Koustav Rudra is a recipient of the DST-INSPIRE Faculty Fellowship [DST/INSPIRE/04/2021/003055] in the year 2021 under Engineering Sciences.

Keywords

  • Ad-hoc document retrieval
  • BERT
  • Distant supervision
  • Label transfer
  • Transfer learning

Fingerprint

Dive into the research topics of 'An in-depth analysis of passage-level label transfer for contextual document ranking'. Together they form a unique fingerprint.

Cite this