Plannable Approximations to MDP Homomorphisms: Equivariance under Actions

Elise van der Pol, Thomas Kipf, Frans A. Oliehoek, Max Welling

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

30 Citations (Scopus)
97 Downloads (Pure)

Abstract

This work exploits action equivariance for representation learning in reinforcement learning. Equivariance under actions states that transitions in the input space are mirrored by equivalent transitions in latent space, while the map and transition functions should also commute. We introduce a contrastive loss function that enforces action equivariance on the learned representations. We prove that when our loss is zero, we have a homomorphism of a deterministic Markov Decision Process (MDP). Learning equivariant maps leads to structured latent spaces, allowing us to build a model on which we plan through value iteration. We show experimentally that for deterministic MDPs, the optimal policy in the abstract MDP can be successfully lifted to the original MDP. Moreover, the approach easily adapts to changes in the goal states. Empirically, we show that in such MDPs, we obtain better representations in fewer epochs compared to representation learning approaches using reconstructions, while generalizing better to new goals than model-free approaches.
Original languageEnglish
Title of host publicationProceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2020
EditorsBo An, Amal El Fallah Seghrouchni, Gita Sukthankar
Place of PublicationRichland, SC
PublisherInternational Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)
Pages1431–1439
Number of pages9
ISBN (Electronic)9781450375184
ISBN (Print)9781450375184
Publication statusPublished - 9 May 2020
EventAAMAS 2020: The 19th International Conference on Autonomous Agents and Multi-Agent Systems - Auckland, New Zealand
Duration: 9 May 202013 May 2020
Conference number: 19th
https://aamas2020.conference.auckland.ac.nz

Publication series

NameProceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
Volume2020-May
ISSN (Print)1548-8403
ISSN (Electronic)1558-2914

Conference

ConferenceAAMAS 2020
Country/TerritoryNew Zealand
CityAuckland
Period9/05/2013/05/20
OtherVirtual/online event due to COVID-19
Internet address

Keywords

  • Equivariance
  • MDP Homomorphisms
  • MDPs
  • Planning

Fingerprint

Dive into the research topics of 'Plannable Approximations to MDP Homomorphisms: Equivariance under Actions'. Together they form a unique fingerprint.

Cite this