Abstract
Reinforcement learning (RL) enables robots to learn skills from interactions with the real world. In practice, the unstructured step-based exploration used in Deep RL – often very successful in simulation – leads to jerky motion patterns on real robots. Consequences of the resulting shaky behavior are poor exploration, or even damage to the robot. We address these issues by adapting state-dependent exploration (SDE) [1] to current Deep RL algorithms. To enable this adaptation, we propose two extensions to the original SDE, using more general features and re-sampling the noise periodically, which leads to a new exploration method generalized state-dependent exploration (gSDE). We evaluate gSDE both in simulation, on PyBullet continuous control tasks, and directly on three different real robots: a tendon-driven elastic robot, a quadruped and an RC car. The noise sampling interval of gSDE enables a compromise between performance and smoothness, which allows training directly on the real robots without loss of performance.
Original language | English |
---|---|
Pages (from-to) | 1634-1644 |
Number of pages | 11 |
Journal | Proceedings of Machine Learning Research |
Volume | 164 |
Publication status | Published - 2021 |
Event | 5th Conference on Robot Learning, CoRL 2021 - London, United Kingdom Duration: 8 Nov 2021 → 11 Nov 2021 |