ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control

Xu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao, Jingkuan Song

November 2026

Clothing folding with pi0.5: the execution length ExRL chooses is plotted next to the rollout.

TL;DR Instead of using RL to change the actions of a long-chunk policy, ExRL keeps the policy frozen and learns only how many actions of each predicted chunk to execute before replanning. On a real arm it lifts a frozen pi0.5 from at most 9/20 to 17/20 on cube stacking, with about ten minutes of interaction.

What they do

Policies such as pi0.5 predict chunks of up to 50 actions. RL methods that adapt them by optimizing whole chunks, or their latent noise, face a search space that grows with the chunk length and need many online samples. ExRL (Execution at the Right Length) leaves the predicted actions alone and learns one discrete decision per query: how many actions of the chunk to execute before asking the policy again.

How it works

Execution-length adapter. A small network takes the current state and the predicted chunk and picks k in {1, …, H}. The base policy stays frozen, so the adapter works with any action-chunked policy.
Duration-aware off-policy RL. The critic Q(s, chunk, k) is trained with a TD target whose accumulated reward and discount depend on k, as in a semi-Markov decision process. ExRL-Q acts greedily on the critic; ExRL-SAC adds a categorical actor. Both use a CQL regularizer.
Online BC (optional). Successful rollouts are distilled back into the base policy, so it can improve beyond what execution length alone allows.
Offline data. Any recorded H-step segment is treated as a virtual chunk whose prefixes become replay transitions for different execution lengths.

Results

LIBERO: pi0.5 goes from nearly 0% to over 90% success within 200,000 environment steps. On Robomimic Tool Hang with Diffusion Policy, ExRL-SAC exceeds 90% while all other baselines stay around 20% or below.
Real AgileX Piper arm, frozen pi0.5 with H=50, ExRL-Q against the best of three fixed execution lengths (1, 10, 50): stack cube 17/20 vs 9/20, clothing folding 17/20 vs 12/20, power insertion 22/25 vs 10/25, parts organization 104/125 vs 69/125 stages.
During training, stacking approaches 100% success after about ten minutes of physical interaction and folding reaches nearly 90% within 85 minutes.
The adapter replans more near contact: mean execution length roughly halves near the object in simulation and falls to about a quarter on real insertion.

Limitations

The real-world runs keep the base policy frozen and do not use online BC, so performance is bounded by what the base policy proposes and several tasks stay below 100%. Each task is adapted separately, with no multi-task or unseen-task evaluation. ExRL has not been tested with asynchronous inference such as real-time chunking.

Videos

Uncut, sped-up ExRL training on stack cube, clothing folding and power insertion, with execution length and success rate.

Figures

From Figure 2: ExRL learns execution lengths with RL, and we optionally use Online BC to distill successful online ro... — From Figure 2: ExRL learns execution lengths with RL, and we optionally use Online BC to distill successful online rollouts back into the base policy during RL.

From Figure 6: ExRL enables sample-efficient online adaptation of different classes of generalist robot policies on L... — From Figure 6: ExRL enables sample-efficient online adaptation of different classes of generalist robot policies on LIBERO, including the flow-based VLA models pi0 and pi0.5, the autoregressive VLA model pi0-FAST, and the world action model Cosmos Policy. All base policies are of chunk size 50.

From Figure 9: How ExRL adapts execution length. (a) Mean execution lengths during free motion and near interaction. ... — From Figure 9: How ExRL adapts execution length. (a) Mean execution lengths during free motion and near interaction. (b) Planned vertical motion over rejected and accepted full chunks on Square. (c) In another Square rollout, short executions progressively align the nut with the peg before ExRL selects a long execution.

From Figure 17: Representative frames and execution-length decisions across four real-world tasks. Stacking and foldi... — From Figure 17: Representative frames and execution-length decisions across four real-world tasks. Stacking and folding use short executions near grasping, followed by longer motions to lift and transport the object or fabric; insertion concentrates short executions near contact, whereas parts organization sustains longer executions across much of the pick-and-place sequence.

Abstract

Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong