Abstract
Deep Reinforcement Learning (RL) is a promising technique towards constructing intelligent agents, but it is not always easy to understand the learning process and the factors that impact it. To shed some light on this, we analyze the Latent State Representations (LSRs) that deep RL agents learn, and compare them to what such agents should ideally learn. We propose a crisp definition of ’ideal LSR’ based on a bisimulation metric, which measures how behaviorally similar states are. The ideal LSR is that in which the distance between two states is proportional to this bisimulation metric. Intuitively, forming such an ideal representation is highly favorable due to its compactness and generalization properties. Here we investigate if this type of representation is also desirable in practice. Our experiments suggest that learning representations that are close to this ideal LSR may improve upon generalization to new irrelevant feature values and modified dynamics. Yet, we show empirically that the extent to which such representations are learned depends on both the network capacity and the state encoding, and that with the current techniques the exact ideal LSR is never formed.
Original language | English |
---|---|
Title of host publication | BNAIC/BeneLearn 2021 |
Subtitle of host publication | 33rd Benelux Conference on Artificial Intelligence and 30th Belgian-Dutch Conference on Machine Learning |
Editors | Edit Luis A. Leiva, Cédric Pruski, Réka Markovich, Amro Najjar, Christoph Schommer |
Pages | 320-334 |
Publication status | Published - 2021 |
Event | 33rd Benelux Conference on Artificial Intelligence and 30th Belgian-Dutch Conference on Machine Learning - Esch-sur-Alzette, Luxembourg Duration: 10 Nov 2021 → 12 Nov 2021 |
Conference
Conference | 33rd Benelux Conference on Artificial Intelligence and 30th Belgian-Dutch Conference on Machine Learning |
---|---|
Abbreviated title | BNAIC/BeneLearn 2021 |
Country/Territory | Luxembourg |
City | Esch-sur-Alzette |
Period | 10/11/21 → 12/11/21 |
Keywords
- Deep Reinforcement Learning
- Bisimulation Metrics