Reinforcement learning of motor skills using Policy Search and human corrective advice

Carlos Celemin, Guilherme Maeda, Javier Ruiz-del-Solar, Jan Peters, Jens Kober

Research output: Contribution to journalArticleScientificpeer-review

2 Citations (Scopus)
175 Downloads (Pure)

Abstract

Robot learning problems are limited by physical constraints, which make learning successful policies for complex motor skills on real systems unfeasible. Some reinforcement learning methods, like Policy Search, offer stable convergence toward locally optimal solutions, whereas interactive machine learning or learning-from-demonstration methods allow fast transfer of human knowledge to the agents. However, most methods require expert demonstrations. In this work, we propose the use of human corrective advice in the actions domain for learning motor trajectories. Additionally, we combine this human feedback with reward functions in a Policy Search learning scheme. The use of both sources of information speeds up the learning process, since the intuitive knowledge of the human teacher can be easily transferred to the agent, while the Policy Search method with the cost/reward function take over for supervising the process and reducing the influence of occasional wrong human corrections. This interactive approach has been validated for learning movement primitives with simulated arms with several degrees of freedom in reaching via-point movements, and also using real robots in such tasks as “writing characters” and the ball-in-a-cup game. Compared with standard reinforcement learning without human advice, the results show that the proposed method not only converges to higher rewards when learning movement primitives, but also that the learning is sped up by a factor of 4–40 times, depending on the task.

Original languageEnglish
Pages (from-to)1560-1580
JournalInternational Journal of Robotics Research
Volume38
Issue number14
DOIs
Publication statusPublished - 2019

Keywords

  • interactive machine learning
  • Learning from Demonstrations
  • motor skills
  • movement primitives
  • Policy Search
  • Reinforcement learning

Fingerprint Dive into the research topics of 'Reinforcement learning of motor skills using Policy Search and human corrective advice'. Together they form a unique fingerprint.

Cite this