Safe Policy Improvement with an Estimated Baseline Policy

Thiago D. Simão, Romain Laroche, Rémi Tachet des Combes

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

5 Downloads (Pure)

Abstract

Previous work has shown the unreliability of existing algorithms in the batch Reinforcement Learning setting, and proposed the theoretically-grounded Safe Policy Improvement with Baseline Bootstrapping (SPIBB) fix: reproduce the baseline policy in the uncertain state-action pairs, in order to control the variance on the trained policy performance. However, in many real-world applications such as dialogue systems, pharmaceutical tests or crop management, data is collected under human supervision and the baseline remains unknown. In this paper, we apply SPIBB algorithms with a baseline estimate built from the data. We formally show safe policy improvement guarantees over the true baseline even without direct access to it. Our empirical experiments on finite and continuous states tasks support the theoretical findings. It shows little loss of performance in comparison with SPIBB when the baseline policy is given, and more importantly, drastically and significantly outperforms competing algorithms both in safe policy improvement, and in average performance.
Original languageEnglish
Title of host publicationProceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems
Place of PublicationRichland, SC
Pages1269–1277
Number of pages9
Publication statusPublished - 2020
EventAAMAS 2020: The 19th International Conference on Autonomous Agents and Multi-Agent Systems - Auckland, New Zealand
Duration: 9 May 202013 May 2020
Conference number: 19th
https://aamas2020.conference.auckland.ac.nz

Publication series

NameAAMAS '20
Publisher International Foundation for Autonomous Agents and Multiagent Systems

Conference

ConferenceAAMAS 2020
CountryNew Zealand
CityAuckland
Period9/05/2013/05/20
OtherVirtual/online event due to COVID-19
Internet address

Fingerprint

Dive into the research topics of 'Safe Policy Improvement with an Estimated Baseline Policy'. Together they form a unique fingerprint.

Cite this