Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems

Miguel Suau; Jinke  He; Matthijs T.J. Spaan; Frans A. Oliehoek

Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems

Miguel Suau^*, Jinke He, Matthijs T.J. Spaan, Frans A. Oliehoek

^*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

26 Downloads (Pure)

Abstract

Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.

Original language	English
Title of host publication	Proceedings of the 39th International Conference on Machine Learning
Editors	K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, S. Sabato
Publisher	PMLR
Pages	20604-20624
Number of pages	21
Volume	162
Publication status	Published - 2022
Event	The 39th International Conference on Machine Learning - Baltimore, United States Duration: 17 Jul 2022 → 23 Jul 2022 Conference number: 39th

Publication series

Name	Proceedings of Machine Learning Research
Volume	162
ISSN (Print)	2640-3498

Conference

Conference	The 39th International Conference on Machine Learning
Abbreviated title	ICML 2022
Country/Territory	United States
City	Baltimore
Period	17/07/22 → 23/07/22

Keywords

reinforcement learning (RL)
simulation and control.

Access to Document

suau22aFinal published version, 1.45 MB

https://proceedings.mlr.press/v162/suau22a.html

Cite this

Suau, M., He, J., Spaan, M. T. J., & Oliehoek, F. A. (2022). Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, & S. Sabato (Eds.), Proceedings of the 39th International Conference on Machine Learning (Vol. 162, pp. 20604-20624). (Proceedings of Machine Learning Research; Vol. 162). PMLR. https://proceedings.mlr.press/v162/suau22a.html

Suau, Miguel ; He, Jinke ; Spaan, Matthijs T.J. et al. / Influence-Augmented Local Simulators : a Scalable Solution for Fast Deep RL in Large Networked Systems. Proceedings of the 39th International Conference on Machine Learning. editor / K. Chaudhuri ; S. Jegelka ; L. Song ; C. Szepesvari ; G. Niu ; S. Sabato. Vol. 162 PMLR, 2022. pp. 20604-20624 (Proceedings of Machine Learning Research).

@inproceedings{ec39b987ac994fa392f4f25006505d50,

title = "Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems",

abstract = "Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.",

keywords = "reinforcement learning (RL), simulation and control.",

author = "Miguel Suau and Jinke He and Spaan, {Matthijs T.J.} and Oliehoek, {Frans A.}",

year = "2022",

language = "English",

volume = "162",

series = "Proceedings of Machine Learning Research",

publisher = "PMLR",

pages = "20604--20624",

editor = "K. Chaudhuri and S. Jegelka and L. Song and C. Szepesvari and G. Niu and S. Sabato",

booktitle = "Proceedings of the 39th International Conference on Machine Learning",

note = "The 39th International Conference on Machine Learning, ICML 2022 ; Conference date: 17-07-2022 Through 23-07-2022",

}

Suau, M , He, J , Spaan, MTJ & Oliehoek, FA 2022, Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems. in K Chaudhuri, S Jegelka, L Song, C Szepesvari, G Niu & S Sabato (eds), Proceedings of the 39th International Conference on Machine Learning. vol. 162, Proceedings of Machine Learning Research, vol. 162, PMLR, pp. 20604-20624, The 39th International Conference on Machine Learning, Baltimore, Maryland, United States, 17/07/22. <https://proceedings.mlr.press/v162/suau22a.html>

Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems. / Suau, Miguel ; He, Jinke ; Spaan, Matthijs T.J. et al.
Proceedings of the 39th International Conference on Machine Learning. ed. / K. Chaudhuri; S. Jegelka; L. Song; C. Szepesvari; G. Niu; S. Sabato. Vol. 162 PMLR, 2022. p. 20604-20624 (Proceedings of Machine Learning Research; Vol. 162).

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Influence-Augmented Local Simulators

T2 - The 39th International Conference on Machine Learning

AU - Suau, Miguel

AU - He, Jinke

AU - Spaan, Matthijs T.J.

AU - Oliehoek, Frans A.

N1 - Conference code: 39th

PY - 2022

Y1 - 2022

N2 - Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.

AB - Learning effective policies for real-world problems is still an open challenge for the field of reinforcement learning (RL). The main limitation being the amount of data needed and the pace at which that data can be obtained. In this paper, we study how to build lightweight simulators of complicated systems that can run sufficiently fast for deep RL to be applicable. We focus on domains where agents interact with a reduced portion of a larger environment while still being affected by the global dynamics. Our method combines the use of local simulators with learned models that mimic the influence of the global system. The experiments reveal that incorporating this idea into the deep RL workflow can considerably accelerate the training process and presents several opportunities for the future.

KW - reinforcement learning (RL)

KW - simulation and control.

M3 - Conference contribution

VL - 162

T3 - Proceedings of Machine Learning Research

SP - 20604

EP - 20624

BT - Proceedings of the 39th International Conference on Machine Learning

A2 - Chaudhuri, K.

A2 - Jegelka, S.

A2 - Song, L.

A2 - Szepesvari, C.

A2 - Niu, G.

A2 - Sabato, S.

PB - PMLR

Y2 - 17 July 2022 through 23 July 2022

ER -

Influence-Augmented Local Simulators: a Scalable Solution for Fast Deep RL in Large Networked Systems

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this