Abstract
This work exploits action equivariance for representation learning in reinforcement learning. Equivariance under actions states that transitions in the input space are mirrored by equivalent transitions in latent space, while the map and transition functions should also commute. We introduce a contrastive loss function that enforces action equivariance on the learned representations. We prove that when our loss is zero, we have a homomorphism of a deterministic Markov Decision Process (MDP). Learning equivariant maps leads to structured latent spaces, allowing us to build a model on which we plan through value iteration. We show experimentally that for deterministic MDPs, the optimal policy in the abstract MDP can be successfully lifted to the original MDP. Moreover, the approach easily adapts to changes in the goal states. Empirically, we show that in such MDPs, we obtain better representations in fewer epochs compared to representation learning approaches using reconstructions, while generalizing better to new goals than model-free approaches.
Original language | English |
---|---|
Title of host publication | Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2020 |
Editors | Bo An, Amal El Fallah Seghrouchni, Gita Sukthankar |
Place of Publication | Richland, SC |
Publisher | International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS) |
Pages | 1431–1439 |
Number of pages | 9 |
ISBN (Electronic) | 9781450375184 |
ISBN (Print) | 9781450375184 |
Publication status | Published - 9 May 2020 |
Event | AAMAS 2020: The 19th International Conference on Autonomous Agents and Multi-Agent Systems - Auckland, New Zealand Duration: 9 May 2020 → 13 May 2020 Conference number: 19th https://aamas2020.conference.auckland.ac.nz |
Publication series
Name | Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS |
---|---|
Volume | 2020-May |
ISSN (Print) | 1548-8403 |
ISSN (Electronic) | 1558-2914 |
Conference
Conference | AAMAS 2020 |
---|---|
Country/Territory | New Zealand |
City | Auckland |
Period | 9/05/20 → 13/05/20 |
Other | Virtual/online event due to COVID-19 |
Internet address |
Keywords
- Equivariance
- MDP Homomorphisms
- MDPs
- Planning