Efficient Exploitation of Factored Domains in Bayesian Reinforcement Learning for POMDPs

Sammie Katt; Frans A. Oliehoek; Christopher Amato

Efficient Exploitation of Factored Domains in Bayesian Reinforcement Learning for POMDPs

Sammie Katt, Frans A. Oliehoek, Christopher Amato

Interactive Intelligence

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

29 Downloads (Pure)

Abstract

While the POMDP has proven to be a powerful framework to model and solve partially observable stochastic problems, it assumes ac- curate and complete knowledge of the environment. When such information is not available, as is the case in many real world appli- cations, one must learn such a model. The BA-POMDP considers the model as part of the hidden state and explicitly considers the uncertainty over it, and as a result transforms the learning problem into a planning problem. This model, however, grows exponentially with the underlying POMDP size, and becomes intractable for non- trivial problems. In this article we propose a factored framework, the FBA-POMDP that represents the model as a Bayes-Net, dras- tically decreasing the number of parameters required to describe the dynamics of the environment. We demonstrate that the our ap- proach allows solvers to tackle problems much larger than possible in the BA-POMDP.

Original language	English
Title of host publication	Adaptive Learning Agents (ALA 2018)
Number of pages	6
Publication status	Published - 1 Jul 2018
Event	ALA 2018 - Workshop at the Federated AI Meeting 2018 - Stockholm, Sweden Duration: 14 Jul 2018 → 15 Jul 2019

Conference

Conference	ALA 2018 - Workshop at the Federated AI Meeting 2018
Abbreviated title	ALA 2018
Country/Territory	Sweden
City	Stockholm
Period	14/07/18 → 15/07/19

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

refereed, workshop

Access to Document

ALA_2018_paper_49-1Final published version, 895 KB

http://ala2018.it.nuigalway.ie/papers/ALA_2018_paper_49.pdf

Cite this

@inproceedings{995e44c1b9b14f15ae3c09bcb51207c9,

title = "Efficient Exploitation of Factored Domains in Bayesian Reinforcement Learning for POMDPs",

abstract = "While the POMDP has proven to be a powerful framework to model and solve partially observable stochastic problems, it assumes ac- curate and complete knowledge of the environment. When such information is not available, as is the case in many real world appli- cations, one must learn such a model. The BA-POMDP considers the model as part of the hidden state and explicitly considers the uncertainty over it, and as a result transforms the learning problem into a planning problem. This model, however, grows exponentially with the underlying POMDP size, and becomes intractable for non- trivial problems. In this article we propose a factored framework, the FBA-POMDP that represents the model as a Bayes-Net, dras- tically decreasing the number of parameters required to describe the dynamics of the environment. We demonstrate that the our ap- proach allows solvers to tackle problems much larger than possible in the BA-POMDP.",

keywords = "refereed, workshop",

author = "Sammie Katt and Oliehoek, {Frans A.} and Christopher Amato",

note = "Green Open Access added to TU Delft Institutional Repository {\textquoteleft}You share, we take care!{\textquoteright} – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.; ALA 2018 - Workshop at the Federated AI Meeting 2018, ALA 2018 ; Conference date: 14-07-2018 Through 15-07-2019",

year = "2018",

month = jul,

day = "1",

language = "English",

booktitle = "Adaptive Learning Agents (ALA 2018)",

}

TY - GEN

T1 - Efficient Exploitation of Factored Domains in Bayesian Reinforcement Learning for POMDPs

AU - Katt, Sammie

AU - Oliehoek, Frans A.

AU - Amato, Christopher

N1 - Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2018/7/1

Y1 - 2018/7/1

N2 - While the POMDP has proven to be a powerful framework to model and solve partially observable stochastic problems, it assumes ac- curate and complete knowledge of the environment. When such information is not available, as is the case in many real world appli- cations, one must learn such a model. The BA-POMDP considers the model as part of the hidden state and explicitly considers the uncertainty over it, and as a result transforms the learning problem into a planning problem. This model, however, grows exponentially with the underlying POMDP size, and becomes intractable for non- trivial problems. In this article we propose a factored framework, the FBA-POMDP that represents the model as a Bayes-Net, dras- tically decreasing the number of parameters required to describe the dynamics of the environment. We demonstrate that the our ap- proach allows solvers to tackle problems much larger than possible in the BA-POMDP.

AB - While the POMDP has proven to be a powerful framework to model and solve partially observable stochastic problems, it assumes ac- curate and complete knowledge of the environment. When such information is not available, as is the case in many real world appli- cations, one must learn such a model. The BA-POMDP considers the model as part of the hidden state and explicitly considers the uncertainty over it, and as a result transforms the learning problem into a planning problem. This model, however, grows exponentially with the underlying POMDP size, and becomes intractable for non- trivial problems. In this article we propose a factored framework, the FBA-POMDP that represents the model as a Bayes-Net, dras- tically decreasing the number of parameters required to describe the dynamics of the environment. We demonstrate that the our ap- proach allows solvers to tackle problems much larger than possible in the BA-POMDP.

KW - refereed, workshop

M3 - Conference contribution

BT - Adaptive Learning Agents (ALA 2018)

T2 - ALA 2018 - Workshop at the Federated AI Meeting 2018

Y2 - 14 July 2018 through 15 July 2019

ER -

Efficient Exploitation of Factored Domains in Bayesian Reinforcement Learning for POMDPs

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Fingerprint

Cite this