Bayesian Reinforcement Learning in Factored POMDPs

Sammie Katt, Frans A. Oliehoek, Christopher Amato

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

3 Citations (Scopus)
37 Downloads (Pure)


Model-based Bayesian Reinforcement Learning (BRL) provides a principled solution to dealing with the exploration-exploitation trade-off, but such methods typically assume a fully observable environments. The few Bayesian RL methods that are applicable in partially observable domains, such as the Bayes-Adaptive POMDP (BA-POMDP), scale poorly. To address this issue, we introduce the Factored BA-POMDP model (FBA-POMDP), a framework that is able to learn a compact model of the dynamics by exploiting the underlying structure of a POMDP. The FBA-POMDP framework casts the problem as a planning task, for which we adapt the Monte-Carlo Tree Search planning algorithm and develop a belief tracking method to approximate the joint posterior over the state and model variables. Our empirical results show that this method outperforms a number of BRL baselines and is able to learn efficiently when the factorization is known, as well as learn both the factorization and the model parameters simultaneously.
Original languageEnglish
Title of host publication18th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2019
Subtitle of host publicationProceedings of the Eighteenth International Conference on Autonomous Agents and Multiagent Systems (AAMAS)
PublisherInternational Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)
Number of pages9
ISBN (Electronic)9781510892002
ISBN (Print)978-1-4503-6309-9
Publication statusPublished - 2019
EventAAMAS 2019: The 18th International Conference on Autonomous Agents and MultiAgent Systems - Montreal, Canada
Duration: 13 May 201917 May 2019

Publication series

NameProceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
ISSN (Print)1548-8403
ISSN (Electronic)1558-2914


ConferenceAAMAS 2019

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.


  • Bayes networks
  • Bayesian Reinforcement Learning
  • Monte-carlo tree search
  • Monte-chain monte-carlo
  • POMDPs


Dive into the research topics of 'Bayesian Reinforcement Learning in Factored POMDPs'. Together they form a unique fingerprint.

Cite this