Graph learning-based generation of abstractions for reinforcement learning

Y. Xue, D. Kudenko, M. Khosla

Research output: Contribution to journalArticleScientificpeer-review

58 Downloads (Pure)

Abstract

The application of reinforcement learning (RL) algorithms is often hindered by the combinatorial explosion of the state space. Previous works have leveraged abstractions which condense large state spaces to find tractable solutions. However, they assumed that the abstractions are provided by a domain expert. In this work, we propose a new approach to automatically construct abstract Markov decision processes (AMDPs) for potential-based reward shaping to improve the sample efficiency of RL algorithms. Our approach to constructing abstract states is inspired by graph representation learning methods, it effectively encodes the topological and reward structure of the ground-level MDP. We perform large-scale quantitative experiments on a range of navigation and gathering tasks under both stationary and stochastic settings. Our approach shows improvements of up to 8.5 times in sample efficiency and up to 3 times in run time over the baseline approach. Besides, with our qualitative analyses of the generated AMDPs, we are able to visually demonstrate the capability of our approach to preserve the topological and reward structure of the ground-level MDP.
Original languageEnglish
Number of pages21
JournalNeural Computing and Applications
DOIs
Publication statusPublished - 2023

Fingerprint

Dive into the research topics of 'Graph learning-based generation of abstractions for reinforcement learning'. Together they form a unique fingerprint.

Cite this