Model-Based Reinforcement Learning with State Abstraction: A Survey

R.A.N. Starre; M. Loog; F.A. Oliehoek

doi:10.1007/978-3-031-39144-6_9

Model-Based Reinforcement Learning with State Abstraction: A Survey

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

78 Downloads (Pure)

Abstract

Model-based reinforcement learning methods are promising since they can increase sample efficiency while simultaneously improving generalizability. Learning can also be made more efficient through state abstraction, which delivers more compact models. Model-based reinforcement learning methods have been combined with learning abstract models to profit from both effects. We consider a wide range of state abstractions that have been covered in the literature, from straightforward state aggregation to deep learned representations, and sketch challenges that arise when combining model-based reinforcement learning with abstraction. We further show how various methods deal with these challenges and point to open questions and opportunities for further research.

Original language	English
Title of host publication	34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn)
Editors	Toon Calders, Bart Goethals, Celine Vens, Jefrey Lijffijt
Chapter	16
Pages	133–148
Number of pages	16
DOIs	https://doi.org/10.1007/978-3-031-39144-6_9
Publication status	Published - 2022
Event	34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn) - Mechelen, Belgium Duration: 7 Nov 2022 → 9 Nov 2022

Publication series

Name	Communications in Computer and Information Science
Volume	1805 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn)
Abbreviated title	BNAIC/BeNeLearn 2022
Country/Territory	Belgium
City	Mechelen
Period	7/11/22 → 9/11/22

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

Model-Based RL
State Abstraction
MDPs

Access to Document

10.1007/978-3-031-39144-6_9

978-3-031-39144-6_9Final published version, 435 KB

Cite this

Starre, R. A. N., Loog, M., & Oliehoek, F. A. (2022). Model-Based Reinforcement Learning with State Abstraction: A Survey. In T. Calders, B. Goethals, C. Vens, & J. Lijffijt (Eds.), 34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn) (pp. 133–148). (Communications in Computer and Information Science; Vol. 1805 CCIS). https://doi.org/10.1007/978-3-031-39144-6_9

Starre, R.A.N. ; Loog, M. ; Oliehoek, F.A. / Model-Based Reinforcement Learning with State Abstraction: A Survey. 34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn). editor / Toon Calders ; Bart Goethals ; Celine Vens ; Jefrey Lijffijt. 2022. pp. 133–148 (Communications in Computer and Information Science).

@inproceedings{1475bd639f2b4bd9989f3e7a160c287a,

title = "Model-Based Reinforcement Learning with State Abstraction: A Survey",

abstract = "Model-based reinforcement learning methods are promising since they can increase sample efficiency while simultaneously improving generalizability. Learning can also be made more efficient through state abstraction, which delivers more compact models. Model-based reinforcement learning methods have been combined with learning abstract models to profit from both effects. We consider a wide range of state abstractions that have been covered in the literature, from straightforward state aggregation to deep learned representations, and sketch challenges that arise when combining model-based reinforcement learning with abstraction. We further show how various methods deal with these challenges and point to open questions and opportunities for further research.",

keywords = "Model-Based RL, State Abstraction, MDPs",

author = "R.A.N. Starre and M. Loog and F.A. Oliehoek",

note = "Green Open Access added to TU Delft Institutional Repository {\textquoteleft}You share, we take care!{\textquoteright} – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. ; 34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn), BNAIC/BeNeLearn 2022 ; Conference date: 07-11-2022 Through 09-11-2022",

year = "2022",

doi = "10.1007/978-3-031-39144-6_9",

language = "English",

isbn = "9783031391439",

series = "Communications in Computer and Information Science",

pages = "133–148",

editor = "Toon Calders and Bart Goethals and Celine Vens and Jefrey Lijffijt",

booktitle = "34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn)",

}

Starre, RAN , Loog, M & Oliehoek, FA 2022, Model-Based Reinforcement Learning with State Abstraction: A Survey. in T Calders, B Goethals, C Vens & J Lijffijt (eds), 34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn). Communications in Computer and Information Science, vol. 1805 CCIS, pp. 133–148, 34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn), Mechelen, Belgium, 7/11/22. https://doi.org/10.1007/978-3-031-39144-6_9

Model-Based Reinforcement Learning with State Abstraction: A Survey. / Starre, R.A.N.; Loog, M.; Oliehoek, F.A.
34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn). ed. / Toon Calders; Bart Goethals; Celine Vens; Jefrey Lijffijt. 2022. p. 133–148 (Communications in Computer and Information Science; Vol. 1805 CCIS).

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Model-Based Reinforcement Learning with State Abstraction: A Survey

AU - Starre, R.A.N.

AU - Loog, M.

AU - Oliehoek, F.A.

N1 - Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2022

Y1 - 2022

N2 - Model-based reinforcement learning methods are promising since they can increase sample efficiency while simultaneously improving generalizability. Learning can also be made more efficient through state abstraction, which delivers more compact models. Model-based reinforcement learning methods have been combined with learning abstract models to profit from both effects. We consider a wide range of state abstractions that have been covered in the literature, from straightforward state aggregation to deep learned representations, and sketch challenges that arise when combining model-based reinforcement learning with abstraction. We further show how various methods deal with these challenges and point to open questions and opportunities for further research.

AB - Model-based reinforcement learning methods are promising since they can increase sample efficiency while simultaneously improving generalizability. Learning can also be made more efficient through state abstraction, which delivers more compact models. Model-based reinforcement learning methods have been combined with learning abstract models to profit from both effects. We consider a wide range of state abstractions that have been covered in the literature, from straightforward state aggregation to deep learned representations, and sketch challenges that arise when combining model-based reinforcement learning with abstraction. We further show how various methods deal with these challenges and point to open questions and opportunities for further research.

KW - Model-Based RL

KW - State Abstraction

KW - MDPs

UR - http://www.scopus.com/inward/record.url?scp=85172085810&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-39144-6_9

DO - 10.1007/978-3-031-39144-6_9

M3 - Conference contribution

SN - 9783031391439

T3 - Communications in Computer and Information Science

SP - 133

EP - 148

BT - 34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn)

A2 - Calders, Toon

A2 - Goethals, Bart

A2 - Vens, Celine

A2 - Lijffijt, Jefrey

T2 - 34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn)

Y2 - 7 November 2022 through 9 November 2022

ER -

Starre RAN , Loog M , Oliehoek FA. Model-Based Reinforcement Learning with State Abstraction: A Survey. In Calders T, Goethals B, Vens C, Lijffijt J, editors, 34th Benelux Conference on Artificial Intelligence (BNAIC) and the 30th Belgian Dutch Conference on Machine Learning (Benelearn). 2022. p. 133–148. (Communications in Computer and Information Science). doi: 10.1007/978-3-031-39144-6_9

Model-Based Reinforcement Learning with State Abstraction: A Survey

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this