Symbolic Regression Methods for Reinforcement Learning

Jiri Kubalik*, Erik Derner, Jan Zegklitz, Robert Babuska

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

7 Citations (Scopus)
74 Downloads (Pure)

Abstract

Reinforcement learning algorithms can solve dynamic decision-making and optimal control problems. With continuous-valued state and input variables, reinforcement learning algorithms must rely on function approximators to represent the value function and policy mappings. Commonly used numerical approximators, such as neural networks or basis function expansions, have two main drawbacks: They are black-box models offering little insight into the mappings learned, and they require extensive trial and error tuning of their hyper-parameters. In this paper, we propose a new approach to constructing smooth value functions in the form of analytic expressions by using symbolic regression. We introduce three off-line methods for finding value functions based on a state-transition model: Symbolic value iteration, symbolic policy iteration, and a direct solution of the Bellman equation. The methods are illustrated on four nonlinear control problems: Velocity control under friction, one-link and two-link pendulum swing-up, and magnetic manipulation. The results show that the value functions yield well-performing policies and are compact, mathematically tractable, and easy to plug into other algorithms. This makes them potentially suitable for further analysis of the closed-loop system. A comparison with an alternative approach using neural networks shows that our method outperforms the neural network-based one.

Original languageEnglish
Pages (from-to)139697-139711
JournalIEEE Access
Volume9
DOIs
Publication statusPublished - 2021

Keywords

  • genetic programming
  • nonlinear optimal control
  • policy iteration
  • Reinforcement learning
  • symbolic regression
  • value iteration

Fingerprint

Dive into the research topics of 'Symbolic Regression Methods for Reinforcement Learning'. Together they form a unique fingerprint.

Cite this