MRHF: Multi-stage Retrieval and Hierarchical Fusion for Textbook Question Answering

Peide Zhu*, Zhen Wang, Manabu Okumura, Jie Yang

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review


Textbook question answering is challenging as it aims to automatically answer various questions on textbook lessons with long text and complex diagrams, requiring reasoning across modalities. In this work, we propose MRHF, a novel framework that incorporates dense passage re-ranking and the mixture-of-experts architecture for TQA. MRHF proposes a novel query augmentation method for diagram questions and then adopts multi-stage dense passage re-ranking with large pretrained retrievers for retrieving paragraph-level contexts. Then it employs a unified question solver to process different types of text questions. Considering the rich blobs and relation knowledge contained in diagrams, we propose to perform multimodal feature fusion over the retrieved context and the heterogeneous diagram features. Furthermore, we introduce the mixture-of-experts architecture to solve the diagram questions to learn from both the rich text context and the complex diagrams and mitigate the possible negative effects between features of the two modalities. We test the framework on the CK12-TQA benchmark dataset, and the results show that MRHF outperforms the state-of-the-art results in all types of questions. The ablation and case study also demonstrates the effectiveness of each component of the framework.

Original languageEnglish
Title of host publicationMultiMedia Modeling - 30th International Conference, MMM 2024, Proceedings
EditorsStevan Rudinac, Marcel Worring, Cynthia Liem, Alan Hanjalic, Björn Pór Jónsson, Yoko Yamakata, Bei Liu
Number of pages14
ISBN (Print)9783031533075
Publication statusPublished - 2024
Event30th International Conference on MultiMedia Modeling, MMM 2024 - Amsterdam, Netherlands
Duration: 29 Jan 20242 Feb 2024

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14555 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference30th International Conference on MultiMedia Modeling, MMM 2024

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.


  • Information Retrieval
  • Mixture-of-Experts
  • Textbook Question Answering


Dive into the research topics of 'MRHF: Multi-stage Retrieval and Hierarchical Fusion for Textbook Question Answering'. Together they form a unique fingerprint.

Cite this