Safe Model-based Reinforcement Learning with Stability Guarantees

Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, Andreas Krause

Seminal May 2017

TL;DR Model-based RL that improves a neural network policy using a Gaussian-process dynamics model and only collects data inside a region of attraction certified by a Lyapunov function. On a simulated inverted pendulum, the certified safe region grows from a small initial set over 50 data points and the pendulum never falls.

What they do

Most RL algorithms explore freely, which can break a real system. Here safety means stability: the state must stay in a region of attraction, from which the policy returns it to the goal. Starting from a policy that is stable only near the goal, the algorithm collects data inside the certified region, learns the dynamics, and updates the policy to enlarge that region and lower the cost.

How it works

Model. A known prior model plus a Gaussian process for the unknown error. Dynamics and policies are Lipschitz, and the model gives high-probability confidence intervals.
Certificate. A level set of a Lyapunov function v is a region of attraction if, on a discretized state grid, the upper confidence bound on v at the next state stays below v(x) minus a Lipschitz margin.
Policy update. Gradient descent on cost-to-go plus a penalty on the worst-case Lyapunov decrease, with the value function as the Lyapunov function. If the region shrinks, the old policy is kept.
Exploration. Sample safe state-actions where the model is most uncertain, with the current policy as backup. Under RKHS assumptions, after finitely many samples this provably explores as far as an oracle that knows the dynamics to ε accuracy.

Results

The test system is a simulated, torque-limited inverted pendulum. The prior model has a lower mass and no friction. The policy is a neural network with two hidden layers of 32 ReLU units.
The initial policy is stable only in a small region. After 50 data points the estimated region of attraction is much larger and lies inside the true safe region. Control performance improves drastically.
Every data point is collected under the current policy, and the pendulum never falls.

Limitations

Grid-based verification scales poorly with state dimension. The method needs a Lyapunov function, an initially safe policy, Lipschitz constants and a deterministic, fully observed system. The practical variant drops the exploration guarantee. Fixing β = 2 makes the guarantee hold per state rather than jointly. Only one simulated pendulum is tested.

Figures

From Figure 1: Initial safe set (red) in a one-dimensional example. Top: the initial policy over state and action. Bo... — From Figure 1: Initial safe set (red) in a one-dimensional example. Top: the initial policy over state and action. Bottom: the Lyapunov function v(x) with its upper and lower confidence bounds u0 and l0.

From Figure 2: Estimated safe set for the inverted pendulum over angle and angular velocity. The initial region of at... — From Figure 2: Estimated safe set for the inverted pendulum over angle and angular velocity. The initial region of attraction V(c0) grows to V(c50) after 50 data points, away from the unsafe region.

Abstract

Reinforcement learning is a powerful paradigm for learning optimal policies from experimental data. However, to find optimal policies, most reinforcement learning algorithms explore all possible actions, which may be harmful for real-world systems. As a consequence, learning algorithms are rarely applied on safety-critical systems in the real world. In this paper, we present a learning algorithm that explicitly considers safety, defined in terms of stability guarantees. Specifically, we extend control-theoretic results on Lyapunov stability verification and show how to use statistical models of the dynamics to obtain high-performance control policies with provable stability certificates. Moreover, under additional regularity assumptions in terms of a Gaussian process prior, we prove that one can effectively and safely collect data in order to learn about the dynamics and thus both improve control performance and expand the safe region of the state space. In our experiments, we show how the resulting algorithm can safely optimize a neural network policy on a simulated inverted pendulum, without the pendulum ever falling down.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong