Influence-aware memory architectures for deep reinforcement learning in POMDPs

Miguel Suau *, Jinke He, Elena Congeduti, Rolf Starre, Aleksander Czechowski, Frans A. Oliehoek

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

3 Citations (Scopus)
17 Downloads (Pure)

Abstract

Due to its perceptual limitations, an agent may have too little information about the environment to act optimally. In such cases, it is important to keep track of the action-observation history to uncover hidden state information. Recent deep reinforcement learning methods use recurrent neural networks (RNN) to memorize past observations. However, these models are expensive to train and have convergence difficulties, especially when dealing with high dimensional data. In this paper, we propose influence-aware memory, a theoretically inspired memory architecture that alleviates the training difficulties by restricting the input of the recurrent layers to those variables that influence the hidden state information. Moreover, as opposed to standard RNNs, in which every piece of information used for estimating Q values is inevitably fed back into the network for the next prediction, our model allows information to flow without being necessarily stored in the RNN’s internal memory. Results indicate that, by letting the recurrent layers focus on a small fraction of the observation variables while processing the rest of the information with a feedforward neural network, we can outperform standard recurrent architectures both in training speed and policy performance. This approach also reduces runtime and obtains better scores than methods that stack multiple observations to remove partial observability.

Original languageEnglish
Number of pages17
JournalNeural Computing and Applications
DOIs
Publication statusPublished - 2022

Keywords

  • Conditional independence
  • Influence
  • Partial observability
  • Recurrent neural networks
  • Reinforcement learning

Fingerprint

Dive into the research topics of 'Influence-aware memory architectures for deep reinforcement learning in POMDPs'. Together they form a unique fingerprint.

Cite this