Unlocking the Flexibility of District Heating Pipeline Energy Storage with Reinforcement Learning

Ksenija Stepanovic*, Jichen Wu, Rob Everhardt, Mathijs de Weerdt

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

4 Citations (Scopus)
69 Downloads (Pure)


The integration of pipeline energy storage in the control of a district heating system can lead to profit gain, for example by adjusting the electricity production of a combined heat and power (CHP) unit to the fluctuating electricity price. The uncertainty from the environment, the computational complexity of an accurate model, and the scarcity of placed sensors in a district heating system make the operational use of pipeline energy storage challenging. A vast majority of previous works determined a control strategy by a decomposition of a mixed-integer nonlinear model and significant simplifications. To mitigate consequential stability, feasibility, and computational complexity challenges, we model CHP economic dispatch as a Markov decision process. We use a reinforcement learning (RL) algorithm to estimate the system’s dynamics through interactions with the simulation environment. The RL approach is compared with a detailed nonlinear mathematical optimizer on day-ahead and real-time electricity markets and two district heating grid models. The proposed method achieves moderate profit impacted by environment stochasticity. The advantages of the RL approach are reflected in three aspects: stability, feasibility, and time scale flexibility. From this, it can be concluded that RL is a promising alternative for real-time control of complex, nonlinear industrial systems.
Original languageEnglish
Article number3290
Pages (from-to)1-25
Number of pages25
Issue number9
Publication statusPublished - 2022


  • 4th generation district heating
  • combined heat and power economic dispatch
  • Markov decision process
  • mixed-integer nonlinear program
  • pipeline energy storage
  • Q-learning


Dive into the research topics of 'Unlocking the Flexibility of District Heating Pipeline Energy Storage with Reinforcement Learning'. Together they form a unique fingerprint.

Cite this