Towards Probabilistically Safe Reinforcement Learning: Theory and Deployment on Humanoid and Aerial Robots

TL;DR Extend a safe-RL method with training-time safety guarantees (RSGF-RL) so it accepts PPO's biased gradient estimates and general trajectory-level constraints. Policies trained with it transfer zero-shot to a Unitree G1 that squats, crawls and walks within height and attitude limits that vanilla PPO breaks, and to a Crazyflie drone that avoids moving obstacles with a 0.1 m safety margin.
What they do
Safe RL methods with guarantees usually handle only expected-cumulative-cost constraints and assume unbiased gradient estimates, which rules out PPO. The authors extend robust safe gradient flow RL (RSGF-RL) to general trajectory-level constraints, including chance constraints, and to biased estimators, and deploy the resulting policies zero-shot on a Unitree G1 humanoid and a Crazyflie drone.
How it works
- Safe update direction. Each step stays as close as possible to the objective’s negative gradient while satisfying a constraint that keeps the safety value feasible, or drives it back if it is not. The direction has a closed form.
- General constraints. A likelihood-ratio identity gives policy gradients for any bounded trajectory-level objective or constraint, such as the probability of staying in a safe set.
- Guarantees under bias. With high probability, constraint violation during training stays within the estimator bias plus a term that shrinks with more rollouts; an unsafe initial policy decays to that level. The iterates have a limit point that is an approximate KKT point.
- PPO in practice. A standard PPO-clip step is kept if it satisfies the constraint, estimated by a learned cost critic, and is otherwise replaced by the safe direction.
Results
- G1 walking (base at least 0.67 m, roll within 0.35 rad, pitch within 0.5 rad, center of pressure in the support polygon): substantially fewer violations than vanilla PPO at comparable final reward.
- Squatting (base at least 0.4 m) and crawling (base and head at most 0.2 m): RSGF-RL learns both motions within the limits; vanilla PPO does not, in simulation or on the robot. On squatting it beats CPO in safety and reward and PPO-Lagrangian in safety at similar reward; a penalty reward matches it only with a task-specific weight.
- Drone (0.1 m margin to obstacles and arena edge): return comparable to vanilla PPO with a low violation rate; on hardware it avoids obstacles and stays in the arena.
Limitations
The experiments break the theory’s rollout-count and stepsize assumptions, so the guarantees are loose, and the biases of PPO and the critic are not tightly bounded. The probabilistic constraint stalled learning, so all experiments use a cumulative cost. Hardware results are shown as example runs without success or violation rates.
Figures


Abstract
Reinforcement learning (RL) enables robots to acquire complex skills, but standard RL formulations do not guarantee that learned policies satisfy safety constraints, which limits their use in safety-critical settings. Existing safe RL methods with provable guarantees typically support only restricted classes of constraints and require unbiased estimators of the objective and constraint functions, which prevents their direct integration with state-of-the-art algorithms such as proximal policy optimization (PPO). In this paper, we extend the recently introduced robust safe gradient flow-based reinforcement learning (RSGF-RL) framework to overcome these limitations. Our formulation accommodates a general class of trajectory-level safety constraints, including chance constraints such as collision avoidance with a prescribed probability, and our analysis allows for biased value and gradient estimators. We show that, with high probability, the constraint violation throughout training is bounded by the estimator bias plus a term that vanishes as the number of rollouts grows, and decays to this level if the initial policy is unsafe. We also establish the convergence properties of the iterates to approximate KKT points of the original safe RL problem. Finally, we show that RSGF-RL implemented with PPO-style estimates enables zero-shot sim-to-real deployment of whole-body control policies on a Unitree G1 humanoid and collision-free waypoint navigation on a Crazyflie drone.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
