← Paper Digest

Towards Pose-Robust Few-Shot Vision-Language-Action Models

Wenhao Li, Zhongyi Han, Yuhao Wei, Liang Han, Yilong Yin, Liqiang Nie

November 2026

Real UR5e rollouts of the four tasks, from lemon to basket to toast into toaster.

TL;DR WACO uses the robot arm, placed by forward kinematics, and the static scene as anchors to estimate camera and base shifts from one RGB-D frame, then maps a few-shot VLA's inputs back to the calibration pose. Under camera and base shifts on a UR5e it raises π0.5's success from 43.3% to 80.0%.

What they do

A VLA fine-tuned on a few dozen demonstrations memorises where its camera and robot base stood during data collection; a bumped tripod or a re-docked base makes it reach for the wrong place. WACO recovers both shifts from one RGB-D frame and maps the observation back to the calibration pose, so an unmodified π0.5 sees what it was trained on. The authors also introduce LIBERO-PoseShift, a benchmark with graded camera, base and combined shifts.

How it works

  • Arm anchor. The arm, placed by forward kinematics, is registered to the camera depth with visibility-aware ICP. This gives the camera-to-base transform, but cannot tell a moved camera from a moved base.
  • World anchor. Matching the scene to an empty-table reference gives the camera shift; with both anchors the base shift follows in closed form. A gate turns canonicalisation off if the implied base motion leaves the floor plane.
  • Canonicalisation. The image is warped with the camera shift; state, actions and start pose are re-expressed with the base shift, turning a moved base into objects moved on the table.
  • Error-aware fine-tuning. Training frames are warped to a random camera and back with the estimator’s error, and states get matching noise. No parameters are added.

Results

  • LIBERO-PoseShift: 87.2% mean shifted success against 74.0% for the standard few-shot recipe, within 1.5 points of a ground-truth-pose oracle. With 10 demonstrations per task WACO (82.4%) beats the standard recipe with 50.
  • LIBERO-Plus camera track: 93.7%, against 62.5% for the public π0.5 checkpoint, with no perturbed training data.
  • UR5e, four tasks: shifted success rises from 43.3% to 80.0% (oracle 85.4%). Median shift errors are 2.8 mm / 0.3° (camera) and 3.8 mm / 0.4° (base).

Limitations

WACO needs an RGB-D third-person camera, a one-off hand-eye calibration, a photo of the empty table and demonstrations that start from one joint configuration. Base shifts are assumed planar. Large camera rotations leave parts of the image unrecoverable, and most remaining LIBERO-Plus failures are at the widest orbits. On the real robot each condition was tested at a single shifted pose.

Figures

From Figure 1: A few-shot fine-tuned VLA misses the target once the camera is bumped or the robot re-docked (left: th...
From Figure 1: A few-shot fine-tuned VLA misses the target once the camera is bumped or the robot re-docked (left: the UR5e setup at the calibration pose and after the table and camera were moved 10 cm away). WACO estimates both shifts from one RGB-D frame and canonicalises the observation back to the calibration viewpoint and base frame, so an unmodified π0.5 sees the world it was trained in (right).
From Figure 2: WACO wraps an unmodified π0.5. At reset, a training-free front end registers the arm model placed by f...
From Figure 2: WACO wraps an unmodified π0.5. At reset, a training-free front end registers the arm model placed by forward kinematics to the observed depth (arm anchor) and matches the static scene to an empty-workspace reference (world anchor); their fusion gives the camera and base shifts and an out-of-plane residual that gates the estimate. Channel A warps the image with the camera shift only; channel B re-expresses state, action and start pose with the base shift only. Bottom left: fine-tuning on the canonicaliser's own artefacts.
From Figure 8: Real-robot success under pose shifts (mean of the camera, base and compound conditions) on the four UR...
From Figure 8: Real-robot success under pose shifts (mean of the camera, base and compound conditions) on the four UR5e tasks. WACO improves every task, most on Toy on Shelf (+46.7 points).
Abstract

Fine-tuning a vision-language-action (VLA) model on a few dozen demonstrations is the standard way to put a manipulation policy on a robot, yet the resulting policy silently assumes that the camera and the robot base stay exactly where they were during data collection: a bumped tripod or a base re-docked a few centimetres away makes it reach for the wrong place. We show that both failures are one phenomenon, a change of the camera-to-base extrinsic that the policy has memorised, and that the robot carries everything needed to undo it. We introduce WACO, a world-anchored canonicalization framework for VLA models: the arm, placed by forward kinematics at the measured joint angles, serves as a metric calibration target, and the static scene serves as the world anchor that tells a moved camera from a moved robot. From a single RGB-D frame, WACO estimates both shifts without ground-truth extrinsics or extra data, canonicalises the image to the calibration viewpoint and the state, action and start pose to the calibration base frame, and fine-tunes the unmodified policy under the artefacts of its own estimator. We also introduce LIBERO-PoseShift, the first benchmark for RGB VLAs with graded camera, base and compound shifts. Across LIBERO-PoseShift, the LIBERO-Plus camera track and four real-world tasks on a UR5e robot, WACO consistently outperforms the standard few-shot recipe and prior viewpoint-robust methods, raising shifted success by 13.2 points in simulation and by 36.7 points on the real robot, recovering 90% of the simulation gap to a ground-truth oracle and 87% of the real-robot gap, and establishing a new state of the art among methods that use no perturbed training data, with no test-time calibration. Project page: https://peacelwh.github.io/WACO/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.