Correct Robots Before They Make Mistakes: Proactive Human-in-the-Loop Intervention via Preference Learning

Xinkai Jiang, Jing Jiang, Hongyi Zhou, Pankhuri Vanjani, Ahmad Baki, Jakub Suliga, Ge Li, Zhuoyue Li, Gerhard Neumann, Rudolf Lioutikov

November 2026

Proactive intervention on Cook the Carrot: the operator takes over before the arm heads to the pot without the carrot.

TL;DR An XR headset shows the operator the policy's predicted gripper path so they can take over before a mistake, and each rejected prediction is paired with the correction that followed for preference post-training. With five correction episodes, mean real-robot success over HG-DAgger rises from 44.0% to 70.3% for a diffusion policy and from 57.8% to 76.9% for X-VLA.

What they do

Human-in-the-loop imitation learning usually collects corrections reactively: the operator takes over after the robot has started to go wrong, so the correction data are full of recovery motions. PHIL (Proactive Human-in-the-Loop) shows the operator the policy’s next action chunk as a 3D gripper path in an XR headset, so they can take over before the mistake. The rejected predictions and the corrections that followed become preference pairs for post-training a diffusion policy or a pretrained VLA (X-VLA).

How it works

XR preview. A Meta Quest 3 overlays a virtual Franka Panda synchronized with the real one and draws the predicted gripper-tip trajectory of each 30-step chunk (queried every 10 steps at 20 Hz). Holding the controller trigger takes over; releasing it hands control back.
Shadow predictions. During takeover the policy keeps running on the same schedule. Each displayed but unexecuted prediction is labeled rejected and paired with the next 30 executed actions from the same state.
Preference post-training. A pairwise logistic loss on the policy’s own imitation loss favors the corrective chunk over the rejected one, alongside imitation on the base demonstrations.

Results

Four real tasks (Pick Up Banana, Towel Folding, Cook the Carrot, Lemon in the Drawer), five correction episodes and 25 evaluation rollouts per condition.

Averaged over tasks and demo budgets, success goes from 44.0% (HG-DAgger, reactive) to 70.3% for the diffusion policy and from 57.8% to 76.9% for X-VLA. Cook the Carrot with 10 demos: 52% vs 24% for HG-DAgger and Residual Policy, 4% for the base policy.
Ablations: proactive data alone gives no consistent gain under behavior cloning; on the same proactive data the preference loss lifts Towel Folding from 40% to 68% and Cook the Carrot from 28% to 52%.
Across 12 HG-DAgger settings, policies trained on proactive corrections show 21 unnecessary recovery behaviors versus 214 for reactive ones.

Limitations

The headline comparison changes both the collection mode and the training objective; only the ablations separate them. Mixing two proactive and three reactive episodes drops PHIL to 0-20% success, below either data source alone. Experiments cover one arm and four tasks, and the recovery analysis counts only successful episodes.

Videos

Cook the Carrot rollouts: HG-DAgger with reactive data, HG-DAgger with proactive data, and PHIL.

Towel Folding rollouts compared across the same three methods.

Figures

From Figure 2: Reactive versus proactive correction. Solid lines are executed motion, dashed lines previewed policy c... — From Figure 2: Reactive versus proactive correction. Solid lines are executed motion, dashed lines previewed policy continuations. Top: reactive intervention follows an observed error and needs recovery. Bottom: the XR preview lets the operator intervene before the error; rejected predictions A- are paired with the executed corrections A+ from the same query state, and control returns to the policy once the preview is acceptable.

From Figure 3: Preference pairs from fixed B-step queries during human intervention. Each rejected H-step prediction ... — From Figure 3: Preference pairs from fixed B-step queries during human intervention. Each rejected H-step prediction A- is paired with the next H executed actions A+, and A+ is labeled preferred. Windows overlap when B is smaller than H, only complete pairs are kept, and the prediction shown at release counts as accepted.

From Figure 4: Proactive intervention on Cook the Carrot. Colored spheres show the predicted end-effector path in XR;... — From Figure 4: Proactive intervention on Cook the Carrot. Colored spheres show the predicted end-effector path in XR; green frames mark policy control, red frames human intervention. The policy moves the pot onto the stove (1-2), the operator takes over when the preview misses the carrot or heads to the pot before grasping it (3-4), and the policy resumes to place the carrot in the pot (5).

From Figure 5: Diffusion Policy performance across base-demonstration budgets: success rates for Pick Up Banana, Towe... — From Figure 5: Diffusion Policy performance across base-demonstration budgets: success rates for Pick Up Banana, Towel Folding, Cook the Carrot and Lemon in the Drawer (a-d), and task completion scores for the two long-horizon tasks (e, f). Each point is 25 rollouts. HG-DAgger and Residual Policy use reactive corrections, PHIL proactive ones, all with five correction episodes.

Abstract

This paper presents Proactive Human-in-the-Loop (PHIL), a framework for proactive policy correction and preference-based post-training. PHIL enables operators to preview policy-predicted future motion and intervene before anticipated undesirable actions. During takeover, PHIL continues querying the policy and displaying its predicted trajectories in the physical workspace through an extended reality (XR) interface. PHIL pairs rejected policy predictions with corrective action chunks subsequently executed from the same physical state over the same horizon, and preference-based post-training encourages the policy to favor these corrections. This paper evaluates PHIL on four real-world manipulation tasks using a diffusion policy and a pretrained vision-language-action policy. With five correction episodes per condition, PHIL increases mean success rates across the evaluated tasks and demonstration budgets from 44.0% to 70.3% for the diffusion policy and from 57.8% to 76.9% for X-VLA, compared with HG-DAgger. Separately, policies trained with proactive corrections show fewer unnecessary recovery behaviors in successful episodes. Project page: https://intuitive-robots.github.io/PHIL-page/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong