TY - GEN
T1 - Safe Policy Improvement with an Estimated Baseline Policy
AU - Simão, Thiago D.
AU - Laroche, Romain
AU - Tachet des Combes, Rémi
N1 - Conference code: 19th
PY - 2020
Y1 - 2020
N2 - Previous work has shown the unreliability of existing algorithms in the batch Reinforcement Learning setting, and proposed the theoretically-grounded Safe Policy Improvement with Baseline Bootstrapping (SPIBB) fix: reproduce the baseline policy in the uncertain state-action pairs, in order to control the variance on the trained policy performance. However, in many real-world applications such as dialogue systems, pharmaceutical tests or crop management, data is collected under human supervision and the baseline remains unknown. In this paper, we apply SPIBB algorithms with a baseline estimate built from the data. We formally show safe policy improvement guarantees over the true baseline even without direct access to it. Our empirical experiments on finite and continuous states tasks support the theoretical findings. It shows little loss of performance in comparison with SPIBB when the baseline policy is given, and more importantly, drastically and significantly outperforms competing algorithms both in safe policy improvement, and in average performance.
AB - Previous work has shown the unreliability of existing algorithms in the batch Reinforcement Learning setting, and proposed the theoretically-grounded Safe Policy Improvement with Baseline Bootstrapping (SPIBB) fix: reproduce the baseline policy in the uncertain state-action pairs, in order to control the variance on the trained policy performance. However, in many real-world applications such as dialogue systems, pharmaceutical tests or crop management, data is collected under human supervision and the baseline remains unknown. In this paper, we apply SPIBB algorithms with a baseline estimate built from the data. We formally show safe policy improvement guarantees over the true baseline even without direct access to it. Our empirical experiments on finite and continuous states tasks support the theoretical findings. It shows little loss of performance in comparison with SPIBB when the baseline policy is given, and more importantly, drastically and significantly outperforms competing algorithms both in safe policy improvement, and in average performance.
UR - http://www.scopus.com/inward/record.url?scp=85096684694&partnerID=8YFLogxK
M3 - Conference contribution
SN - 9781450375184
T3 - AAMAS '20
SP - 1269
EP - 1277
BT - Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems
CY - Richland, SC
T2 - AAMAS 2020
Y2 - 9 May 2020 through 13 May 2020
ER -