Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

OopsieRecover: Learning Transferable VLA Corrections from Semantic Interventions

Arpit Bahety, Skand Peri, Sriniket Ambatipudi, Chen Xu, Haruki Nishimura, Masha Itkina, Roberto Martín-Martín

November 2026

Real robot, steered pi0.5 picking up a coffee pod and loading it into the machine.

TL;DR OopsieRecover turns a handful of human interventions, each with a short note on why, into gated costs over 3D point sets that decide when and how to steer a frozen pi0.5 VLA. On a real DROID setup it raises task progress from 35.1% for the next-best method to 61.0%, and the corrections carry over to new object poses and scenes.

What they do

VLAs fail in systematic ways, and DAgger-style finetuning needs many corrections that mostly fix only the states where they were collected. OopsieRecover asks the human for a correction plus a sentence on why. A VLM turns a few of these into a reusable rule, a gated cost saying when the correction applies and what to optimize. At test time the rule steers a frozen pi0.5 by choosing among its own sampled action chunks.

How it works

  • Grounding. A VLM names the entities involved (mug, machine panel, tray) and marks a seed point on each; segmentation and depth turn them into tracked 3D point sets.
  • Abduction. From the point-set trajectories before and during the takeover, the VLM writes a cost with a gate, a correction term and a task term. The gate threshold is set to the lowest score seen just before the human stepped in.
  • Verification. The cost must rate the pre-intervention behavior worse than the human’s and must fire before the takeover, for every correction of that type; otherwise the VLM revises it.
  • Steering. When a gate fires, sampled VLA chunks are rolled out with a dynamics model (simulator physics in sim, a grasp-based model on the robot) and the lowest-cost one runs.

Results

  • Real robot (DROID setup; Put Croffle, Load Coffee Pod, Shelve Item; 3-6 corrections per task; 20 rollouts per level): task progress of 61.0% vs 35.1% for the next-best method. It beats HG-DAgger* by at least 60% at every level and is the only top method at L0 and L2, tying pi0.5 at L1.
  • Simulation (three RoboCasa tasks, 5-8 corrections, 50 episodes): HG-DAgger* improves only in the reset state where corrections were collected, while OopsieRecover is alone at the top for new poses and new scenes.
  • On CoffeeMug, 84% success after 8 corrections, against 28% for HG-DAgger*.
  • One “avoid the steam handle” correction triggered and prevented the induced failure in 10 of 10 novel scenes.

Limitations

Corrections must be spatial conditions over point sets, with no contact forces or hidden physical properties. Results depend on grounding, tracking and motion prediction. Steering can only pick from the sampled chunks, and verification checks consistency with recorded interventions, not correctness everywhere.

Videos

Collecting a correction, with the base policy running until a human takes over.
After five rounds of corrections, the learned corrections trigger and steer the base policy.

Figures

From Figure 1: A human corrects a lid-closure failure and describes the intended behavior. OopsieRecover grounds this...
From Figure 1: A human corrects a lid-closure failure and describes the intended behavior. OopsieRecover grounds this feedback in semantic point sets and infers a gated correction cost that steers a frozen VLA. Right: the same correction programs are reused under changed object poses and scene layouts.
From Figure 2: A human monitors the VLA and intervenes with a corrective trajectory and a language description (orang...
From Figure 2: A human monitors the VLA and intervenes with a corrective trajectory and a language description (orange). The interventions of each correction type become a semantically grounded gated cost: point sets are tracked on scene entities, costs are generated from their motion and verified against all corrections of that type (blue). At deployment (green) the costs decide when to steer and rank candidate actions, while the VLA stays frozen.
From Figure 4: Success in simulation (top) and task progress on the real robot (bottom) at difficulty levels L0 to L2...
From Figure 4: Success in simulation (top) and task progress on the real robot (bottom) at difficulty levels L0 to L2, pooled over three tasks. Letters mark statistically separated groups. OopsieRecover leads or ties at every level.
From Figure 5: Success on CoffeeMug against the number of corrections. OopsieRecover reaches 84% with 8 corrections, ...
From Figure 5: Success on CoffeeMug against the number of corrections. OopsieRecover reaches 84% with 8 corrections, while HG-DAgger* reaches 28%.
Abstract

Vision-Language-Action (VLA) models can perform a wide range of robotic manipulation tasks, but often fail during deployment. Teaching them to recover from these failures requires substantial corrective data: existing approaches learn to recover from failures by directly associating corrective actions with the observations in which they were demonstrated. For instance, DAgger-like fine-tuning can require tens to hundreds of interactions, yet the resulting policy may still fail to generalize the learned corrections beyond the demonstrated scenarios. We hypothesize that semantics can capture the underlying reasons for failures and their corrections, enabling policies to learn from only a handful of corrective demonstrations and generalize those corrections to novel objects, and scene variations. We propose OopsieRecover, a framework that uses abduction to infer the intent behind a small set of corrective demonstrations and encodes the inferred knowledge as dense spatial representations and conditional costs. These costs capture when a behavior is undesirable and steer the policy toward more desirable action regions. We experiment across 6 tasks and variations in object poses, scenes, and related tasks, and find that only a handful of corrective demonstrations can enable a VLA to overcome failures across a broad range of novel situations, without requiring new corrective data for each variation. Please refer to https://robin-lab.cs.utexas.edu/oopsierecover for videos.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.