← Paper Digest

VG-Steer: Visual-Geometric Policy Steering from In-Context Aligned Abstractions

Arthur Bucker, Pramod Anantharam, Jonathan Francis, Jean Oh

November 2026

Real UR5 with live segmentation of the cable and bag, grasping the cable under guidance.

TL;DR VG-Steer corrects a frozen flow-matching policy at test time by steering each denoising step toward the distances between gripper and objects recorded in one retrieved successful rollout. It lifts π0.5 from 26.3% to 37.5% on shifted LIBERO-Pro tasks, and on a real UR5 raises near-out-of-distribution cable bagging from 8/27 to 17/27.

What they do

Fine-tuned visuomotor policies break under small shifts in where objects are or which object is named. VG-Steer adapts a frozen flow-matching policy at test time, with no new data and no weight updates. It takes one successful in-distribution rollout and reduces it to a few time-varying distances between the gripper and task-relevant objects. It then steers the policy’s denoising so the predicted action chunk reproduces those distances in the new scene.

How it works

  • Retrieve and abstract. The closest rollout is retrieved by DINOv2 image and CLIP text similarity, then split into phases at low end-effector speed. For each phase, a VLM (GPT-5.4) picks entity pairs and a projection axis or plane. A pair can be a positional distance, the gripper’s orientation against an axis, or the gripper aperture.
  • Ground. Gemini 3 Flash boxes refined by SAM2 masks label the objects in both the demonstration and the live scene. Depth gives object centroids and surface points.
  • Steer. At every denoising step, a differentiable model predicts the distances a candidate chunk would produce: an integrator of velocity actions in simulation, UR5 forward kinematics on hardware. The normalized gradient of the deviation from the time-aligned reference is applied with a golden-section line search. If no step reduces the loss, the unsteered step is kept.

Results

  • LIBERO-Pro task and positional shifts with π0.5: 37.5% average, against 26.3% unsteered and 29.8% for the reproduced VLS steering baseline. The same guidance lifts GR00T N1.7 from 9.9% to 11.8%. Overhead is about 12 ms per call (1.2x), against about 20x for full VLS.
  • Ablation: guidance with a raw fixed-step gradient is worse than none (23.8%). The line search lifts it to 35.0%, and normalizing the gradient gives the full 37.5%.
  • Real UR5: a π0.5 fine-tuned on 28 cable-bagging demos is steered from one of its own rollouts. Positional success rises from 35.4% to 54.2%. When the prompt names a banana or a spoon instead, task success rises from 33.3% to 62.5% and wrong-object grasps fall from 45.8% to 4.2%.

Limitations

Simulation results use ground-truth segmentation. With real open-vocabulary perception, VG-Steer drops to 24.8% and 24.5% on task and positional shifts, so perception is the main source of error. Mis-detected sub-goals can make phase switching advance too early, stall or oscillate. Far out-of-distribution placements are not recovered. Once the approach is corrected, the remaining failures come from grasping and transport, where over-steering can disrupt the policy’s motor skills.

Videos

Real-robot positional sweep, with the cable placed in each of the 16 grid cells.
Animation of object-relational distance pairs from a cube, re-targeted to a cylinder.

Figures

From Figure 1: Offline, once per episode: (1) retrieve the most relevant successful in-distribution rollout by image ...
From Figure 1: Offline, once per episode: (1) retrieve the most relevant successful in-distribution rollout by image (DINOv2) and text (CLIP) similarity; (2) a VLM selects task-relevant distance pairs per motion phase (e.g. gripper-cube distance, cube-table distance, gripper orientation), giving reference trends; (3) a VLM with SAM2 matches the demonstration objects to the new scene (the cylinder takes the cube's role). Online, every action chunk: (4) perception aligns the live distances to the reference, and at every denoising step a line-searched gradient step on the predicted distances corrects the frozen policy's velocity field.
From Figure 2: At each denoising step the candidate chunk is rolled out by forward kinematics and the guidance loss (...
From Figure 2: At each denoising step the candidate chunk is rolled out by forward kinematics and the guidance loss (position, orientation, gripper) is differentiated. A golden-section line search sets the step size, and the correction is applied to the policy's velocity field without modifying weights.
From Figure 6: Real-robot cable bagging success against the cable's distance from the training positions, with a logi...
From Figure 6: Real-robot cable bagging success against the cable's distance from the training positions, with a logistic fit per method. Both succeed 9/9 in distribution and 0/12 far out of distribution; in the near band guidance raises success from 8/27 to 17/27 and moves the 50% crossing from 2.8 to 3.4 standard deviations.
From Figure 9: A cable-only policy is asked to bag the cable, banana or spoon with all three on the table. Unsteered,...
From Figure 9: A cable-only policy is asked to bag the cable, banana or spoon with all three on the table. Unsteered, it often reaches the wrong object; with VG-Steer it reaches the named object in 23 of 24 trials.
Abstract

Adapting robot policies to out-of-distribution (OOD) settings with limited real-world data remains a key bottleneck to generalist robots. The multimodality and scarcity of robotics data lead end-to-end approaches to overfit and break under small distribution shifts. Test-time adaptation can alleviate this by correcting frozen policies at inference, but most methods ground their correction in a signal learned implicitly from robotic datasets, making it susceptible to the same shifts as the policy it corrects. Recent methods that instead exploit explicit spatial information are typically limited to coarse corrections, missing the nuanced, task-relevant motion most manipulation tasks require. We propose VG-Steer, which corrects existing policies by transferring the nuanced motion of successful in-distribution rollouts, captured along meaningful geometric axes, known here as object-relational distance pairs. VG-Steer leverages the semantic understanding of Large Vision Models (LVMs) and Vision-Language Models (VLMs) reasoning, which combines the task-relevant entity pairs and their object-relational distance projections into a single tailored visual-geometric guidance signal. In a simulation benchmark, it efficiently corrects flow-matching policies under pose and task-goal shifts; on a real robot experiment, it widens the region of the workspace the frozen policy covers, more than doubling its success rate on cable placements one grid step outside the training positions, and steers a single-task policy toward objects it was never trained on while nearly eliminating object confusion. Project website: https://arthurfenderbucker.github.io/VG-Steer/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.