Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators

Miguel Suau; Jinke He; Matthijs T.J. Spaan; Frans A. Oliehoek

Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators

Miguel Suau^*, Jinke He, Matthijs T.J. Spaan, Frans A. Oliehoek

^*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

15 Downloads (Pure)

Abstract

Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.

Original language	English
Title of host publication	International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022
Publisher	International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)
Pages	1735-1737
Number of pages	3
ISBN (Electronic)	978-171385433-3
Publication status	Published - 2022
Event	21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022 - Auckland, Virtual, New Zealand Duration: 9 May 2022 → 13 May 2022

Publication series

Name	Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
Volume	3
ISSN (Print)	1548-8403
ISSN (Electronic)	1558-2914

Conference

Conference	21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022
Country/Territory	New Zealand
City	Auckland, Virtual
Period	9/05/22 → 13/05/22

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

Deep Reinforcement Learning
Influence
Simulation

Access to Document

3535850.3536093Final published version, 1.34 MB

Cite this

Suau, M., He, J., Spaan, M. T. J., & Oliehoek, F. A. (2022). Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators. In International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022 (pp. 1735-1737). (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS; Vol. 3). International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

Suau, Miguel ; He, Jinke ; Spaan, Matthijs T.J. et al. / Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators. International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022. pp. 1735-1737 (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS).

@inproceedings{0307bd8246f04387a80553bf44c572b0,

title = "Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators",

abstract = "Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.",

keywords = "Deep Reinforcement Learning, Influence, Simulation",

author = "Miguel Suau and Jinke He and Spaan, {Matthijs T.J.} and Oliehoek, {Frans A.}",

note = "Green Open Access added to TU Delft Institutional Repository {\textquoteleft}You share, we take care!{\textquoteright} – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. ; 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022 ; Conference date: 09-05-2022 Through 13-05-2022",

year = "2022",

language = "English",

series = "Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS",

publisher = "International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)",

pages = "1735--1737",

booktitle = "International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022",

}

Suau, M , He, J , Spaan, MTJ & Oliehoek, FA 2022, Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators. in International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, vol. 3, International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), pp. 1735-1737, 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, Virtual, New Zealand, 9/05/22.

Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators. / Suau, Miguel ; He, Jinke ; Spaan, Matthijs T.J. et al.
International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022. p. 1735-1737 (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS; Vol. 3).

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators

AU - Suau, Miguel

AU - He, Jinke

AU - Spaan, Matthijs T.J.

AU - Oliehoek, Frans A.

N1 - Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2022

Y1 - 2022

N2 - Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.

AB - Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.

KW - Deep Reinforcement Learning

KW - Influence

KW - Simulation

UR - http://www.scopus.com/inward/record.url?scp=85134338671&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85134338671

T3 - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS

SP - 1735

EP - 1737

BT - International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022

PB - International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)

T2 - 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022

Y2 - 9 May 2022 through 13 May 2022

ER -

Suau M , He J , Spaan MTJ , Oliehoek FA. Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators. In International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022. International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS). 2022. p. 1735-1737. (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS).

Speeding up Deep Reinforcement Learning through Influence-Augmented Local Simulators

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this