Policy derivation methods for critic-only reinforcement learning in continuous spaces

Eduard Alibekov, Jiri Kubalik, Robert Babuska

Research output: Contribution to journalArticleScientificpeer-review

7 Citations (Scopus)
9 Downloads (Pure)

Abstract

This paper addresses the problem of deriving a policy from the value function in the context of critic-only reinforcement learning (RL) in continuous state and action spaces. With continuous-valued states, RL algorithms have to rely on a numerical approximator to represent the value function. Numerical approximation due to its nature virtually always exhibits artifacts which damage the overall performance of the controlled system. In addition, when continuous-valued action is used, the most common approach is to discretize the action space and exhaustively search for the action that maximizes the right-hand side of the Bellman equation. Such a policy derivation procedure is computationally involved and results in steady-state error due to the lack of continuity. In this work, we propose policy derivation methods which alleviate the above problems by means of action space refinement, continuous approximation, and post-processing of the V-function by using symbolic regression. The proposed methods are tested on nonlinear control problems: 1-DOF and 2-DOF pendulum swing-up problems, and on magnetic manipulation. The results show significantly improved performance in terms of cumulative return and computational complexity.
Original languageEnglish
Pages (from-to)178-187
JournalEngineering Applications of Artificial Intelligence
Volume69
DOIs
Publication statusPublished - 2018

Bibliographical note

Accepted Author Manuscript

Keywords

  • Reinforcement learning
  • Continuous actions
  • Multi-variable systems
  • Optimal control
  • Policy derivation
  • Optimization

Fingerprint

Dive into the research topics of 'Policy derivation methods for critic-only reinforcement learning in continuous spaces'. Together they form a unique fingerprint.

Cite this