GSBR: Geometric Support Behavior Regularization for Online RL

Charles Xu, Jacob Berg, Liyiming Ke, Sergey Levine

November 2026

RoboMimic tool-hang in simulation, one of the four RoboMimic tasks.

TL;DR GSBR fine-tunes a frozen imitation-learned base policy with online RL, penalizing actions only by how far they fall outside the base policy's estimated support. Across 19 simulated tasks on four benchmarks it raises average success from 33% for the base policy to 95%, with about 23% higher normalized throughput than the next-best RL method.

What they do

Online RL on top of a frozen diffusion or flow-matching policy is usually regularized toward it. Behavior-cloning penalties stop the actor from dropping poor modes; hard support constraints forbid anything just outside the base policy’s actions. GSBR charges nothing inside the base policy’s support and a penalty that grows with distance outside it, using only samples from the base policy.

How it works

Penalty. Zero inside the support and the squared distance to the nearest supported action outside it, so the actor leaves the prior only when the critic’s value gain outweighs the cost.
Support from samples. Draw N action chunks from the base policy, set a radius to a quantile q of their nearest-neighbor distances, and take the union of balls of that radius around the samples. The approximation error vanishes as N grows.
Actor-critic. A Gaussian chunk actor conditioned on the state and the reference actions, trained against 10 critics on a 50/50 mix of offline and online data.

Results

19 simulated tasks (RoboMimic, OGBench cube-triple, OCBench, and LIBERO-90 with a π0.5 base policy from pixels): average success rises from 33% for the base policy to 95%, against 92% for the next best (RLT). Normalized throughput is 0.95 against 0.77.
OGBench cube-triple, where the base policy never succeeds: 99% average success; on task 5, 95% against 58% for the best baseline.
Ablations on cube-triple: 8 reference samples suffice, and a zero radius drops normalized throughput from 1.00 to 0.66.

Limitations

Simulation only; the authors name real-robot validation as the next step. Sampling the base policy at test time adds compute and latency, and the quantile q is an extra hyperparameter, swept per task together with the penalty weight.

Videos

OGBench cube-triple, a long-horizon task where the base policy succeeds 0% of the time.

A LIBERO-90 kitchen task from pixels, with π0.5 as the base policy.

Figures

From Figure 1: Left: (a) A behavior-cloning regularizer matches the base policy distribution, so the RL actor cannot ... — From Figure 1: Left: (a) A behavior-cloning regularizer matches the base policy distribution, so the RL actor cannot drop poor modes. (b) A support constraint lets the actor select the best behavior in the base policy distribution but excludes improvements beyond it. (c) GSBR penalizes distance outside the base policy support, so the actor can discard suboptimal behaviors and explore beyond the prior. Right: normalized throughput across 19 tasks and 4 benchmarks, 23% higher for GSBR than for previous methods.

From Figure 2: GSBR. Draw samples from the base policy and compute the nearest-neighbor distances between them. Set t... — From Figure 2: GSBR. Draw samples from the base policy and compute the nearest-neighbor distances between them. Set the radius to a quantile of these distances and build the support region as the union of balls of that radius around each sample. The behavior regularizer is the distance to the closest point of that region.

From Figure 4: Normalized throughput across benchmarks. GSBR has the highest average over all 19 tasks and leads all ... — From Figure 4: Normalized throughput across benchmarks. GSBR has the highest average over all 19 tasks and leads all baselines on RoboMimic, OGBench and OCBench. Throughput (success rate divided by mean episode length) is normalized by the best method per task, then averaged across tasks.

Abstract

Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies -- typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets -- capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy's support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy's support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance while still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Project website and open-source code at https://charlesxu0124.github.io/gsbr/ .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong