Abstract
Deep reinforcement learning makes it possible to train control policies that map high-dimensional observations to actions. These methods typically use gradient-based optimization techniques to enable relatively efficient learning, but are notoriously sensitive to hyperparameter choices and do not have good convergence properties. Gradient-free optimization methods, such as evolutionary strategies, can offer a more stable alternative but tend to be much less sample efficient. In this work we propose a combination, using the relative strengths of both. We start with a gradient-based initial training phase, which is used to quickly learn both a state representation and an initial policy. This phase is followed by a gradient-free optimization of only the final action selection parameters. This enables the policy to improve in a stable manner to a performance level not obtained by gradient-based optimization alone, using many fewer trials than methods using only gradient-free optimization. We demonstrate the effectiveness of the method on two Atari games, a continuous control benchmark and the CarRacing-v0 benchmark. On the latter we surpass the best previously reported score while using significantly fewer episodes.
Original language | English |
---|---|
Pages (from-to) | 8049-8056 |
Journal | IFAC-PapersOnline |
Volume | 53 |
Issue number | 2 |
DOIs | |
Publication status | Published - 2020 |
Event | 21st IFAC World Congress 2020 - Berlin, Germany Duration: 12 Jul 2020 → 17 Jul 2020 |
Keywords
- Control
- Deep learning
- Neural networks
- Optimization
- Reinforcement learning