Zero-Shot VLA Adaptation with Retrieved Guidance

Subin Kim, Anh-Quan Pham, Dinesh Jayaraman

November 2026

Real Franka rollout with the retrieved guidance trajectory drawn over the scene.

TL;DR RAGu retrieves demonstrations from mismatched datasets such as DROID, warps them into the target scene and uses them to steer a frozen VLA's flow-matching denoising, with no training. Real-world success of pi0.5 rises from 20% to 80%.

What they do

Vision-language-action models degrade under small shifts in viewpoint, background, objects or embodiment. RAGu brings retrieval-augmented generation to VLAs without retraining: it pulls demonstrations of the same task category from a large, mismatched dataset (DROID or MetaWorld), warps them into the target scene, and uses the best one to steer the frozen policy’s action denoising.

How it works

Filter and warp. Demonstrations are filtered by task keywords such as “pick” and “place”. Grounding DINO boxes locate the task-relevant objects in the target scene, and each source trajectory’s waypoints are shifted onto them. Warping runs offline.
Coarse-to-fine retrieval. Dynamic time warping against one reference trajectory prunes the pool to the top N. At inference, the candidate whose cost gradient has the smallest norm is selected, i.e. the one that requires the least steering away from what the VLA already intends.
Guidance. The selected Cartesian trajectory defines an energy added to the flow-matching velocity. Forward kinematics maps actions to end-effector paths, so no inverse kinematics or joint mapping is needed across embodiments.

Results

Real Franka, 5 tasks with 10 rollouts each: pi0.5 20%, Omniguide 40%, RICL 0%, RAGu 80%. On Wipe Table RAGu reaches 100% against 30% for Omniguide.
RoboLab simulation, 3 tasks: 25.3% for RAGu against 5.3% for pi0.5 and 12% for VLS. Over nine tasks, mean success goes from 23.6% unguided to 28.4% with one retrieved trajectory and 36.4% with a pool of 100.
Sim-to-real from MetaWorld Sawyer demonstrations: 60% on the Cabinet to Shelf layout, where pi0.5 and Omniguide get 0%.
Franka demonstrations guiding GR00T-N1.5 on a GR-1 humanoid in RoboCasa: +13.6 points on average, +12.0 for the fine-tuned model.

Limitations

The source database must contain demonstrations of roughly the target task category. The method is not invariant to camera placement and occlusion, since correspondence failures corrupt the warp; in RoboLab 34.9% of warping attempts fail. Guidance can only reshape motion within the VLA’s prior.

Videos

Another real-world rollout, a tabletop task with bowls, with the guidance trajectory overlaid.

Real-world rollout on a white tabletop, with the guidance trajectory overlaid.

Figures

From Figure 2: RAGu transfers manipulation knowledge across four settings, real to real, real to sim, sim to real and... — From Figure 2: RAGu transfers manipulation knowledge across four settings, real to real, real to sim, sim to real and across morphologies, with the base VLA's weights frozen.

From Figure 1: The RAGu pipeline. (a) Demonstrations are filtered by task-category keywords. (b) Task-relevant object... — From Figure 1: The RAGu pipeline. (a) Demonstrations are filtered by task-category keywords. (b) Task-relevant objects in the target scene are found by semantic bounding-box detection, and source trajectories are warped onto them. (c) Candidates are pruned to the top N by dynamic time warping, then the one with the smallest guidance gradient norm is selected. (d) At test time the chosen trajectory defines an energy field that steers denoising.

From Figure 5: Rollouts on the Wipe Table task. RAGu guides the end-effector along the table surface for a successful... — From Figure 5: Rollouts on the Wipe Table task. RAGu guides the end-effector along the table surface for a successful wipe. Omniguide lifts the towel off the surface on the way to its next waypoint, and RICL fails.

From Figure 8: Breakdown of warping outcomes in RoboLab. Demo parsing errors (22.0%) come from source demonstrations ... — From Figure 8: Breakdown of warping outcomes in RoboLab. Demo parsing errors (22.0%) come from source demonstrations whose gripper events do not fit the target task. Scene correspondence errors (10.4%) come from triangulation failure, cross-view inconsistency or out-of-bounds waypoints. These are warping outcomes, not task success rates.

Abstract

Pre-training large parametric "Vision-Language-Action" (VLA) models on large demonstration datasets has recently yielded promising results for general-purpose manipulation. However, these models still struggle under even small domain shifts relative to the training data, such as changes in camera viewpoint, background, object identity, or robot morphology. Motivated by the success of Retrieval-Augmented Generation (RAG) in language models, we pursue retrieval to boost VLA performance. Specifically, we seek to leverage retrieval from domain-mismatched, "in-the-wild" demonstration datasets like DROID to guide a VLA's behavior in a new target domain. To achieve this, we introduce RAGu, which overcomes robotics-specific challenges for RAG: (1) It incorporates simple geometric calibration to transform source demonstrations into the target setup and reduce cross-domain differences. (2) It uses a coarse-to-fine retrieval strategy that selects, from a large pool of pre-warped candidates, the trajectory most compatible with the base VLA. (3) It guides the VLA's flow-matching inference procedure rather than seeking to retrain the base model or expand its limited context window. We evaluate RAGu across real-world, simulation, sim-to-real transfer, and cross-embodiment humanoid manipulation. We find it improves real-world VLA success rates from 20% to 80% and outperforms the closest-related baselines in sim. Project website: raguvla.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong