Beyond Policy Patches: Distilling RL Specialists into Vision-Language-Action Policies

Zhennan Jiang, Tingying Xie, Shilong Ji, Shu'ang Yu, Tianxing Zhou, Yuhui Chen, Jiahuan Gao, Kaizhi Xie, Yuanfan Xu, Feng Gao, Yu Wang, Haoran Li, Dongbin Zhao, Chao Yu

November 2026

Dual-arm peg insertion on the real bimanual Franka setup.

TL;DR REPAIR distills phase-specific RL specialists into a single pi0.5 VLA by training it on critic-weighted specialist action candidates, first on specialist rollouts and then on states the VLA reaches itself. On six real bimanual Franka tasks it succeeds in 30/30 trials on every task, with 29.8-67.5% lower end-effector jerk than keeping the specialists in the loop.

What they do

VLA policies often handle the easy stages of a long-horizon task but fail in short precision-critical phases such as insertion, grasping or handover. Small RL specialists can fix those phases, but they stay in the deployed system as extra controllers with handoffs, and their execution is jittery. REPAIR moves the specialists’ skill into the VLA’s own action expert, so only the VLA is deployed.

How it works

Base and specialists. pi0.5 is fine-tuned on human teleoperation demos, then frozen while a lightweight MLP actor and critic (RL Token) is trained on the real robot, with human interventions, for each annotated critical phase.
Value-refined targets. At each critical-phase observation the phase’s specialist proposes 8 action chunks. The critic scores them, the scores are normalized within the group, and a softmax turns them into weights for a weighted flow-matching loss. Noncritical observations imitate stored VLA actions.
Offline, then on-policy. Training starts on observations from 10 specialist-assisted rollouts, then continues on 10 rollouts run by the VLA alone. There the specialists label states but never take control, and the critics keep updating.
Every method gets 50 adaptation rollouts per task; REPAIR spends 30 of them training the specialist.

Results

Six real-robot tasks on two Franka arms, 30 trials each: REPAIR and specialist takeover (RLT) both reach 100% on all six. Base SFT gets 0-33.3%, and EXPO-FT and ConRFT get 0% on bagging, table clearing and drawer stowing.
On the four long-horizon tasks HG-DAgger averages 74.2% success, against 100% for REPAIR.
Mean jerk is 29.8-67.5% lower than RLT on every task. On dual-arm peg insertion the mean episode length drops from 249.7 to 238.0.
Ablation on dual peg (offline only): cloning the specialist’s executed actions reaches 60%, eight critic-weighted candidates 90%.

Limitations

Training needs experimenter-supplied phase annotations, and the critic’s rankings on learner-visited states are not directly validated. HG-DAgger has slightly lower mean jerk on four tasks. All tasks run on one bimanual platform.

Videos

Grocery bagging on the real robot.

Bimanual table clearing, with bowls handed between arms into a drying basket.

Figures

From Figure 1: Overview of REPAIR. Local RL expertise provides value-guided supervision for offline initialization an... — From Figure 1: Overview of REPAIR. Local RL expertise provides value-guided supervision for offline initialization and on-policy repair. The VLA acts without specialist takeover during online distillation and is deployed as a single policy.

From Figure 4: Training framework of REPAIR. (a) Human demonstrations initialize the base VLA pi0.5. (b) With this VL... — From Figure 4: Training framework of REPAIR. (a) Human demonstrations initialize the base VLA pi0.5. (b) With this VLA frozen, critical-phase RL trains specialist actors and their critics. (c) Offline and on-policy observations feed a shared value-refined supervision mechanism: the selected phase specialist generates action candidates and its critic weights them. The VLA learns from these targets in critical phases and replayed VLA actions in noncritical phases. Only the VLA is deployed.

From Figure 6: Dual-arm peg insertion episode lengths for REPAIR and RLT (30 trials per method; 100% success).

Abstract

Vision-language-action (VLA) policies can handle noncritical stages of long-horizon manipulation yet fail during short precision-critical phases. Lightweight reinforcement learning (RL) specialists can address these failures without directly applying RL to the full VLA, but retain auxiliary controllers and switching mechanisms at deployment. We study how to transfer their capabilities into a single flow-matching VLA. Direct imitation of specialist rollouts can reproduce action jitter and leave the VLA unprepared for states encountered during its own execution. We introduce REPAIR, a value-guided distillation framework that combines offline initialization with on-policy refinement. At each critical-phase observation, REPAIR queries the corresponding specialist for multiple action candidates and weights them using its critic to construct a supervision distribution. A shared weighted flow-matching objective trains the VLA first on observations from specialist-assisted rollouts, then on observations collected through its own execution. Specialists provide supervision without taking control during on-policy collection and are removed at deployment. Across six real-robot tasks with 30 evaluation trials per task, REPAIR achieves 100% success on all six tasks, with lower mean end-effector jerk than specialist-assisted execution on every task.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong