Safe Reinforcement Learning via Shielding

Abstract

Reinforcement learning algorithms discover policies that maximize reward, but do not necessarily guarantee safety during learning or execution phases. We introduce a new approach to learn optimal policies while enforcing properties expressed in temporal logic. To this end, given the temporal logic specification that is to be obeyed by the learning system, we propose to synthesize a reactive system called a shield. The shield is introduced in the traditional learning process in two alternative ways, depending on the location at which the shield is implemented. In the first one, the shield acts each time the learning agent is about to make a decision and provides a list of safe actions. In the second way, the shield is introduced after the learning agent. The shield monitors the actions from the learner and corrects them only if the chosen action causes a violation of the specification. We discuss which requirements a shield must meet to preserve the convergence guarantees of the learner. Finally, we demonstrate the versatility of our approach on several challenging reinforcement learning scenarios.

DarcStar commentary

Our perspective on this work and how it informs our research — not part of the original paper.

This is the same instinct as a control barrier function, expressed in the language of logic and automata rather than dynamics: put a shield between the learner and the world that corrects any action about to violate a specification. The learner explores freely; the shield vetoes what's unsafe. Same shape we favor — capability and safety as separate, composable pieces.

The detail we appreciate is that the shield is synthesized from a specification, not learned. You write the property you want preserved in temporal logic, and the shield is provably correct with respect to it — and the paper is careful about the requirement that the shield preserve the learner's convergence, so safety doesn't come at the cost of never learning anything. That's the right kind of rigor: a safety layer that is auditable and doesn't quietly break the thing it wraps.

Where our emphasis sits is the specification itself. A shield is exactly as trustworthy as the property it enforces, and the truly hard work — the work no synthesis procedure does for you — is writing down a specification that actually captures what you meant by “safe.” The mechanism is solved here; the semantics are the standing problem, and they're a human one.