Configuration of the Actor and Critic Network of the Deep Reinforcement Learning controller for Multi-Energy Storage System

Paula  Páramo-Balsa; Francisco Gonzalez-Longatt; Martha N.  Acosta; José Luis Rueda Torres; Peter Palensky; Francisco Sanchez; Juan Manuel Roldan-Fernandez; Manuel  Burgos-Payán

doi:10.1109/GPECOM55404.2022.9815793

Configuration of the Actor and Critic Network of the Deep Reinforcement Learning controller for Multi-Energy Storage System

Paula Páramo-Balsa, Francisco Gonzalez-Longatt, Martha N. Acosta , José Luis Rueda Torres, Peter Palensky, Francisco Sanchez, Juan Manuel Roldan-Fernandez, Manuel Burgos-Payán

Intelligent Electrical Power Grids

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

21 Downloads (Pure)

Abstract

The computational burden and the time required to train a deep reinforcement learning (DRL) can be appreciable, especially for the particular case of a DRL control used for frequency control of multi-electrical energy storage (MEESS). This paper presents an assessment of four training configurations of the actor and critic network to determine the configuration training that produces the lower computational time, considering the specific case of frequency control of MEESS. The training configuration cases are defined considering two processing units: CPU and GPU and are evaluated considering serial and parallel computing using MATLAB® 2020b Parallel Computing Toolbox. The agent used for this assessment is the Deep Deterministic Policy Gradient (DDPG) agent. The environment represents the dynamic model to provide enhanced frequency response to the power system by controlling the state of charge of energy storage systems. Simulation results demonstrated that the best configuration to reduce the computational time is training both actor and critic network on CPU using parallel computing.

Original language	English
Title of host publication	Proceedings of the 2022 4th Global Power, Energy and Communication Conference (GPECOM)
Place of Publication	Piscataway
Publisher	IEEE
Pages	564-568
Number of pages	5
ISBN (Electronic)	978-1-6654-6925-8
ISBN (Print)	978-1-6654-6926-5
DOIs	https://doi.org/10.1109/GPECOM55404.2022.9815793
Publication status	Published - 2022
Event	2022 4th Global Power, Energy and Communication Conference (GPECOM) - Nevsehir, Turkey Duration: 14 Jun 2022 → 17 Jun 2022 Conference number: 4th

Conference

Conference	2022 4th Global Power, Energy and Communication Conference (GPECOM)
Country/Territory	Turkey
City	Nevsehir
Period	14/06/22 → 17/06/22

Bibliographical note

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care

Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

actor-network
critic network
deep reinforcement learning
energy storage systems
enhanced frequency response
parallel computing

Access to Document

10.1109/GPECOM55404.2022.9815793

Configuration_of_the_Actor_and_Critic_Network_of_the_Deep_Reinforcement_Learning_controller_for_Multi-Energy_Storage_SystemFinal published version, 1.01 MB

Cite this

Páramo-Balsa, P., Gonzalez-Longatt, F., Acosta , M. N., Rueda Torres, J. L., Palensky, P., Sanchez, F., Roldan-Fernandez, J. M., & Burgos-Payán, M. (2022). Configuration of the Actor and Critic Network of the Deep Reinforcement Learning controller for Multi-Energy Storage System. In Proceedings of the 2022 4th Global Power, Energy and Communication Conference (GPECOM) (pp. 564-568). Article 9815793 IEEE. https://doi.org/10.1109/GPECOM55404.2022.9815793

@inproceedings{5ddd951e87084961b95e12b0c5427d03,

title = "Configuration of the Actor and Critic Network of the Deep Reinforcement Learning controller for Multi-Energy Storage System",

abstract = "The computational burden and the time required to train a deep reinforcement learning (DRL) can be appreciable, especially for the particular case of a DRL control used for frequency control of multi-electrical energy storage (MEESS). This paper presents an assessment of four training configurations of the actor and critic network to determine the configuration training that produces the lower computational time, considering the specific case of frequency control of MEESS. The training configuration cases are defined considering two processing units: CPU and GPU and are evaluated considering serial and parallel computing using MATLAB{\textregistered} 2020b Parallel Computing Toolbox. The agent used for this assessment is the Deep Deterministic Policy Gradient (DDPG) agent. The environment represents the dynamic model to provide enhanced frequency response to the power system by controlling the state of charge of energy storage systems. Simulation results demonstrated that the best configuration to reduce the computational time is training both actor and critic network on CPU using parallel computing.",

keywords = "actor-network, critic network, deep reinforcement learning, energy storage systems, enhanced frequency response, parallel computing",

author = "Paula P{\'a}ramo-Balsa and Francisco Gonzalez-Longatt and Acosta, {Martha N.} and {Rueda Torres}, {Jos{\'e} Luis} and Peter Palensky and Francisco Sanchez and Roldan-Fernandez, {Juan Manuel} and Manuel Burgos-Pay{\'a}n",

note = "Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.; 2022 4th Global Power, Energy and Communication Conference (GPECOM) ; Conference date: 14-06-2022 Through 17-06-2022",

year = "2022",

doi = "10.1109/GPECOM55404.2022.9815793",

language = "English",

isbn = "978-1-6654-6926-5",

pages = "564--568",

booktitle = "Proceedings of the 2022 4th Global Power, Energy and Communication Conference (GPECOM)",

publisher = "IEEE",

address = "United States",

}

Páramo-Balsa, P, Gonzalez-Longatt, F, Acosta , MN, Rueda Torres, JL , Palensky, P, Sanchez, F, Roldan-Fernandez, JM & Burgos-Payán, M 2022, Configuration of the Actor and Critic Network of the Deep Reinforcement Learning controller for Multi-Energy Storage System. in Proceedings of the 2022 4th Global Power, Energy and Communication Conference (GPECOM)., 9815793, IEEE, Piscataway, pp. 564-568, 2022 4th Global Power, Energy and Communication Conference (GPECOM), Nevsehir, Turkey, 14/06/22. https://doi.org/10.1109/GPECOM55404.2022.9815793

Configuration of the Actor and Critic Network of the Deep Reinforcement Learning controller for Multi-Energy Storage System. / Páramo-Balsa, Paula ; Gonzalez-Longatt, Francisco; Acosta , Martha N. et al.
Proceedings of the 2022 4th Global Power, Energy and Communication Conference (GPECOM). Piscataway: IEEE, 2022. p. 564-568 9815793.

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Configuration of the Actor and Critic Network of the Deep Reinforcement Learning controller for Multi-Energy Storage System

AU - Páramo-Balsa, Paula

AU - Gonzalez-Longatt, Francisco

AU - Acosta , Martha N.

AU - Rueda Torres, José Luis

AU - Palensky, Peter

AU - Sanchez, Francisco

AU - Roldan-Fernandez, Juan Manuel

AU - Burgos-Payán, Manuel

N1 - Conference code: 4th

PY - 2022

Y1 - 2022

N2 - The computational burden and the time required to train a deep reinforcement learning (DRL) can be appreciable, especially for the particular case of a DRL control used for frequency control of multi-electrical energy storage (MEESS). This paper presents an assessment of four training configurations of the actor and critic network to determine the configuration training that produces the lower computational time, considering the specific case of frequency control of MEESS. The training configuration cases are defined considering two processing units: CPU and GPU and are evaluated considering serial and parallel computing using MATLAB® 2020b Parallel Computing Toolbox. The agent used for this assessment is the Deep Deterministic Policy Gradient (DDPG) agent. The environment represents the dynamic model to provide enhanced frequency response to the power system by controlling the state of charge of energy storage systems. Simulation results demonstrated that the best configuration to reduce the computational time is training both actor and critic network on CPU using parallel computing.

AB - The computational burden and the time required to train a deep reinforcement learning (DRL) can be appreciable, especially for the particular case of a DRL control used for frequency control of multi-electrical energy storage (MEESS). This paper presents an assessment of four training configurations of the actor and critic network to determine the configuration training that produces the lower computational time, considering the specific case of frequency control of MEESS. The training configuration cases are defined considering two processing units: CPU and GPU and are evaluated considering serial and parallel computing using MATLAB® 2020b Parallel Computing Toolbox. The agent used for this assessment is the Deep Deterministic Policy Gradient (DDPG) agent. The environment represents the dynamic model to provide enhanced frequency response to the power system by controlling the state of charge of energy storage systems. Simulation results demonstrated that the best configuration to reduce the computational time is training both actor and critic network on CPU using parallel computing.

KW - actor-network

KW - critic network

KW - deep reinforcement learning

KW - energy storage systems

KW - enhanced frequency response

KW - parallel computing

UR - http://www.scopus.com/inward/record.url?scp=85134875263&partnerID=8YFLogxK

U2 - 10.1109/GPECOM55404.2022.9815793

DO - 10.1109/GPECOM55404.2022.9815793

M3 - Conference contribution

SN - 978-1-6654-6926-5

SP - 564

EP - 568

BT - Proceedings of the 2022 4th Global Power, Energy and Communication Conference (GPECOM)

PB - IEEE

CY - Piscataway

T2 - 2022 4th Global Power, Energy and Communication Conference (GPECOM)

Y2 - 14 June 2022 through 17 June 2022

ER -

Páramo-Balsa P, Gonzalez-Longatt F, Acosta MN, Rueda Torres JL , Palensky P, Sanchez F et al. Configuration of the Actor and Critic Network of the Deep Reinforcement Learning controller for Multi-Energy Storage System. In Proceedings of the 2022 4th Global Power, Energy and Communication Conference (GPECOM). Piscataway: IEEE. 2022. p. 564-568. 9815793 doi: 10.1109/GPECOM55404.2022.9815793

Configuration of the Actor and Critic Network of the Deep Reinforcement Learning controller for Multi-Energy Storage System

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this