Impact of Pre-training on Deep Reinforcement Learning Ramp Metering Systems

C. Evans; M. Rinaldi; Henk Taale; S.P. Hoogendoorn

Impact of Pre-training on Deep Reinforcement Learning Ramp Metering Systems

C. Evans^*, M. Rinaldi, Henk Taale, S.P. Hoogendoorn

^*Corresponding author for this work

Traffic Systems Engineering

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

Abstract

Pre-training is a process used to enhance the learning of deep reinforcement learning (RL)
algorithms through initial guidance from an expert demonstrator. This involves training a neural
network to replicate the outputs of the selected expert before allowing the RL agent to specialise and develop its own policy. This paper outlines a study that aims to analyse the impact of pre-training on deep RL algorithms used in ramp metering. Specifically, behaviour cloning is performed for increasing lengths of time (0-10,000 epochs), with ALINEA as the chosen expert algorithm guiding a proposed Proximal Policy Optimisation (PPO)-based system. The results confirm that, with the same length of training, some initial guidance through pre-training can significantly improve the
system’s effectiveness in reducing congestion compared to no pre-training. Otherwise, excessive
pre-training may lead to overfitting and reduced generalisability. Design issues resulting in weak
model convergence, however, limit the algorithm’s overall performance in the chosen scenario.

Original language	English
Title of host publication	Proceedings of the National Academy of Science’s Transportation Research Board 104th Annual Meeting
Publication status	Unpublished - 7 Jan 2025
Event	104th Annual Meeting of the Transportation Research Board (TRB) - Walter E. Washington Convention Center, Washington DC, United States Duration: 5 Jan 2025 → 9 Jan 2025 https://trb-annual-meeting.nationalacademies.org/schedule

Conference

Conference	104th Annual Meeting of the Transportation Research Board (TRB)
Abbreviated title	TRB 2025
Country/Territory	United States
City	Washington DC
Period	5/01/25 → 9/01/25
Internet address	https://trb-annual-meeting.nationalacademies.org/schedule

Keywords

Network management
Road traffic control
Ramp metering
Reinforcement learning

Country (case study)

Netherlands

TUD-SUMO
Evans, C. (Creator), GitHub, 16 Jul 2024
https://github.com/DAIMoNDLab/tud-sumo and one more link, https://pypi.org/project/tud-sumo/ (show fewer)
Dataset/Software: Software

Cite this

@inproceedings{3b31b2cc18964cb2aa6afa10dd451674,

title = "Impact of Pre-training on Deep Reinforcement Learning Ramp Metering Systems",

abstract = "Pre-training is a process used to enhance the learning of deep reinforcement learning (RL)algorithms through initial guidance from an expert demonstrator. This involves training a neuralnetwork to replicate the outputs of the selected expert before allowing the RL agent to specialise and develop its own policy. This paper outlines a study that aims to analyse the impact of pre-training on deep RL algorithms used in ramp metering. Specifically, behaviour cloning is performed for increasing lengths of time (0-10,000 epochs), with ALINEA as the chosen expert algorithm guiding a proposed Proximal Policy Optimisation (PPO)-based system. The results confirm that, with the same length of training, some initial guidance through pre-training can significantly improve thesystem{\textquoteright}s effectiveness in reducing congestion compared to no pre-training. Otherwise, excessivepre-training may lead to overfitting and reduced generalisability. Design issues resulting in weakmodel convergence, however, limit the algorithm{\textquoteright}s overall performance in the chosen scenario.",

keywords = "Network management, Road traffic control, Ramp metering, Reinforcement learning",

author = "C. Evans and M. Rinaldi and Henk Taale and S.P. Hoogendoorn",

year = "2025",

month = jan,

day = "7",

language = "English",

booktitle = "Proceedings of the National Academy of Science{\textquoteright}s Transportation Research Board 104th Annual Meeting",

note = "104th Annual Meeting of the Transportation Research Board (TRB), TRB 2025 ; Conference date: 05-01-2025 Through 09-01-2025",

url = "https://trb-annual-meeting.nationalacademies.org/schedule",

}

TY - GEN

T1 - Impact of Pre-training on Deep Reinforcement Learning Ramp Metering Systems

AU - Evans, C.

AU - Rinaldi, M.

AU - Taale, Henk

AU - Hoogendoorn, S.P.

PY - 2025/1/7

Y1 - 2025/1/7

N2 - Pre-training is a process used to enhance the learning of deep reinforcement learning (RL)algorithms through initial guidance from an expert demonstrator. This involves training a neuralnetwork to replicate the outputs of the selected expert before allowing the RL agent to specialise and develop its own policy. This paper outlines a study that aims to analyse the impact of pre-training on deep RL algorithms used in ramp metering. Specifically, behaviour cloning is performed for increasing lengths of time (0-10,000 epochs), with ALINEA as the chosen expert algorithm guiding a proposed Proximal Policy Optimisation (PPO)-based system. The results confirm that, with the same length of training, some initial guidance through pre-training can significantly improve thesystem’s effectiveness in reducing congestion compared to no pre-training. Otherwise, excessivepre-training may lead to overfitting and reduced generalisability. Design issues resulting in weakmodel convergence, however, limit the algorithm’s overall performance in the chosen scenario.

AB - Pre-training is a process used to enhance the learning of deep reinforcement learning (RL)algorithms through initial guidance from an expert demonstrator. This involves training a neuralnetwork to replicate the outputs of the selected expert before allowing the RL agent to specialise and develop its own policy. This paper outlines a study that aims to analyse the impact of pre-training on deep RL algorithms used in ramp metering. Specifically, behaviour cloning is performed for increasing lengths of time (0-10,000 epochs), with ALINEA as the chosen expert algorithm guiding a proposed Proximal Policy Optimisation (PPO)-based system. The results confirm that, with the same length of training, some initial guidance through pre-training can significantly improve thesystem’s effectiveness in reducing congestion compared to no pre-training. Otherwise, excessivepre-training may lead to overfitting and reduced generalisability. Design issues resulting in weakmodel convergence, however, limit the algorithm’s overall performance in the chosen scenario.

KW - Network management

KW - Road traffic control

KW - Ramp metering

KW - Reinforcement learning

M3 - Conference contribution

BT - Proceedings of the National Academy of Science’s Transportation Research Board 104th Annual Meeting

T2 - 104th Annual Meeting of the Transportation Research Board (TRB)

Y2 - 5 January 2025 through 9 January 2025

ER -

Impact of Pre-training on Deep Reinforcement Learning Ramp Metering Systems

Abstract

Conference

Keywords

Country (case study)

Fingerprint

Datasets

TUD-SUMO

Cite this