VG-Steer: Visual-Geometric Policy Steering from In-Context Aligned Abstractions
TL;DR VG-Steer corrects a frozen flow-matching policy at test time by steering each denoising step toward the distances between gripper and objects recorded in one retrieved successful rollout. It lifts π0.5 from 26.3% to 37.5% on shifted LIBERO-Pro tasks, and on a real UR5 raises near-out-of-distribution cable bagging from 8/27 to 17/27.
What they do
Fine-tuned visuomotor policies break under small shifts in where objects are or which object is named. VG-Steer adapts a frozen flow-matching policy at test time, with no new data and no weight updates. It takes one successful in-distribution rollout and reduces it to a few time-varying distances between the gripper and task-relevant objects. It then steers the policy’s denoising so the predicted action chunk reproduces those distances in the new scene.
How it works
- Retrieve and abstract. The closest rollout is retrieved by DINOv2 image and CLIP text similarity, then split into phases at low end-effector speed. For each phase, a VLM (GPT-5.4) picks entity pairs and a projection axis or plane. A pair can be a positional distance, the gripper’s orientation against an axis, or the gripper aperture.
- Ground. Gemini 3 Flash boxes refined by SAM2 masks label the objects in both the demonstration and the live scene. Depth gives object centroids and surface points.
- Steer. At every denoising step, a differentiable model predicts the distances a candidate chunk would produce: an integrator of velocity actions in simulation, UR5 forward kinematics on hardware. The normalized gradient of the deviation from the time-aligned reference is applied with a golden-section line search. If no step reduces the loss, the unsteered step is kept.
Results
- LIBERO-Pro task and positional shifts with π0.5: 37.5% average, against 26.3% unsteered and 29.8% for the reproduced VLS steering baseline. The same guidance lifts GR00T N1.7 from 9.9% to 11.8%. Overhead is about 12 ms per call (1.2x), against about 20x for full VLS.
- Ablation: guidance with a raw fixed-step gradient is worse than none (23.8%). The line search lifts it to 35.0%, and normalizing the gradient gives the full 37.5%.
- Real UR5: a π0.5 fine-tuned on 28 cable-bagging demos is steered from one of its own rollouts. Positional success rises from 35.4% to 54.2%. When the prompt names a banana or a spoon instead, task success rises from 33.3% to 62.5% and wrong-object grasps fall from 45.8% to 4.2%.
Limitations
Simulation results use ground-truth segmentation. With real open-vocabulary perception, VG-Steer drops to 24.8% and 24.5% on task and positional shifts, so perception is the main source of error. Mis-detected sub-goals can make phase switching advance too early, stall or oscillate. Far out-of-distribution placements are not recovered. Once the approach is corrected, the remaining failures come from grasping and transport, where over-steering can disrupt the policy’s motor skills.
Videos
Figures




Abstract
Adapting robot policies to out-of-distribution (OOD) settings with limited real-world data remains a key bottleneck to generalist robots. The multimodality and scarcity of robotics data lead end-to-end approaches to overfit and break under small distribution shifts. Test-time adaptation can alleviate this by correcting frozen policies at inference, but most methods ground their correction in a signal learned implicitly from robotic datasets, making it susceptible to the same shifts as the policy it corrects. Recent methods that instead exploit explicit spatial information are typically limited to coarse corrections, missing the nuanced, task-relevant motion most manipulation tasks require. We propose VG-Steer, which corrects existing policies by transferring the nuanced motion of successful in-distribution rollouts, captured along meaningful geometric axes, known here as object-relational distance pairs. VG-Steer leverages the semantic understanding of Large Vision Models (LVMs) and Vision-Language Models (VLMs) reasoning, which combines the task-relevant entity pairs and their object-relational distance projections into a single tailored visual-geometric guidance signal. In a simulation benchmark, it efficiently corrects flow-matching policies under pose and task-goal shifts; on a real robot experiment, it widens the region of the workspace the frozen policy covers, more than doubling its success rate on cable placements one grid step outside the training positions, and steers a single-task policy toward objects it was never trained on while nearly eliminating object confusion. Project website: https://arthurfenderbucker.github.io/VG-Steer/
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
