Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement Learning

Thomas M. Moerland, Anna Deichler, Simone Baldi, Joost Broekens, Catholijn M. Jonker

Research output: Chapter in Book/Conference proceedings/Edited volumeChapterScientificpeer-review

18 Downloads (Pure)

Abstract

Planning and reinforcement learning are two key approaches to sequential decision making. Multi-step approximate real-time dynamic programming, a recently successful algorithm class of which AlphaZero [Silver et al., 2018] is an example, combines both by nesting planning within a learning loop. However, the combination of planning and learning introduces a new question: how should we balance time spend on planning, learning and acting? The importance of this trade-off has not been explicitly studied before. We show that it is actually of key importance, with computational results indicating that we should neither plan too long nor too short. Conceptually, we identify a new spectrum of planning-learning algorithms which ranges from exhaustive search (long planning) to model-free RL (no planning), with optimal performance achieved midway.
Original languageEnglish
Title of host publicationICAPS: PRL 2020
Subtitle of host publicationProceedings of the 1st Workshop on Bridging the Gap Between AI Planning and Reinforcement Learning (PRL)
EditorsAlan Fern, Vicenc Gomez, Anders Jonsson, Michael Katz, Hector Palacios, Scott Sanner
PublisherAssociation for the Advancement of Artificial Intelligence (AAAI)
Pages53-60
Number of pages8
Publication statusPublished - 2020
EventICAPS 2020: 30th International Conference on Automated Planning and Scheduling - Virtual Nancy, France
Duration: 19 Oct 202030 Oct 2020
Conference number: 30th

Conference

ConferenceICAPS 2020
Country/TerritoryFrance
CityVirtual Nancy
Period19/10/2030/10/20

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Fingerprint

Dive into the research topics of 'Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement Learning'. Together they form a unique fingerprint.

Cite this