CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration

Q. Yang*, M.T.J. Spaan*

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

1 Downloads (Pure)


Without an assigned task, a suitable intrinsic objective for an agent is to explore the environment efficiently. However, the pursuit of exploration will inevitably bring more safety risks.
An under-explored aspect of reinforcement learning is how to achieve safe efficient exploration when the task is unknown.
In this paper, we propose a practical Constrained Entropy Maximization (CEM) algorithm to solve task-agnostic safe exploration problems, which naturally require a finite horizon and undiscounted constraints on safety costs.
The CEM algorithm aims to learn a policy that maximizes the state entropy under the premise of safety.
To avoid approximating the state density in complex domains, CEM leverages a $k$-nearest neighbor entropy estimator to evaluate the efficiency of exploration.
In terms of safety, CEM minimizes the safety costs, and adaptively trades off safety and exploration based on the current constraint satisfaction. We empirically show that CEM allows learning a safe exploration policy in complex continuous-control domains, and the learned policy benefits downstream tasks in safety and sample efficiency.
Original languageEnglish
Title of host publicationThe Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI-23)
Publication statusAccepted/In press - 2023
Event37th AAAI Conference on Artificial Intelligence - Washington, United States
Duration: 7 Feb 202314 Feb 2023
Conference number: 37


Conference37th AAAI Conference on Artificial Intelligence
Abbreviated titleAAAI-23
Country/TerritoryUnited States


  • Reinforcement Learning
  • Safe Exploration


Dive into the research topics of 'CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration'. Together they form a unique fingerprint.

Cite this