VICTR: Value-Informed In-Context Learning for Vision-Language-Action Models

Luca Macesanu, Tianluo Zhang, Aditya Patil, Archit Sharma, Kevin Angers, Christopher M. Clark, Chen Feng, Juexiao Zhang

November 2026

Unseen task "put the square on the peg": reference demo next to VICTR, π0.5 and vision-retrieval rollouts.

TL;DR VICTR feeds a π0.5 policy the demonstration chunk at the same stage of the task, picked by a progress estimator that reads a reference demo, instead of the most visually similar chunk. On unseen real-robot tasks success rises from 6.2% for fine-tuned π0.5 (15.0% with visual retrieval) to 23.8%.

What they do

Retrieval-based in-context learning puts a chunk of a stored demonstration into a VLA’s context. The VLA can then adapt to a new task from a few demos without fine-tuning. Earlier work (RICL) retrieves the most visually similar chunk, but similar-looking frames can come from different stages of a task. VICTR retrieves by estimated task progress instead.

How it works

IC-VFE. A progress estimator built from Gemma 3 4B’s frozen vision encoder and a Gemma 3 270M decoder. It reads the current observation, the instruction and a full reference demo compressed to 16 tokens per view. It outputs progress in [0,1] as a distribution over 201 bins. Training labels are pseudo-labels from the Robo-Dopamine reward model.
Retrieval. Each task has a pool of 10 demos split into one-second chunks, each scored by IC-VFE; the shortest demo is the reference. At every step the chunk with the closest progress value is retrieved.
IC-VLA. π0.5 post-trained so that the retrieved chunk’s images, state and FAST action tokens precede the current query under block-causal attention.
Robot. Bimanual YOR robot with two AgileX Piper arms, a head camera and two wrist cameras; 28 tasks, 7.5 h of data.

Results

On 8 unseen tasks (10 rollouts each), success goes from 6.2% (fine-tuned π0.5) and 15.0% (visual retrieval) to 23.8%, and progress from 28.9% and 42.1% to 55.5%. Adding visual similarity to the retrieval signal does not help (21.2%).
On 8 seen tasks, all retrieval methods reach 87.5-90% progress versus 82.2% for π0.5.
Against human-annotated progress on unseen tasks, IC-VFE scores Pearson r 0.65 vs 0.57 for RECAP; without the reference demo it drops to 0.49.
On 30 unseen LIBERO-100 tasks: 45.9% vs 37.8% (π0.5) and 41.2% (RICL).

Limitations

Success on unseen tasks stays low. All data and tests come from one physical environment. Squeezing a full demo into the estimator’s context costs spatial detail.

Videos

Unseen task "uncap the black marker": reference demo next to VICTR, π0.5 and vision-retrieval rollouts.

Figures

From Figure 1: Value-informed context retrieval for in-context robot learning. Given a reference demonstration, IC-VF... — From Figure 1: Value-informed context retrieval for in-context robot learning. Given a reference demonstration, IC-VFE estimates task-relative progress, which VICTR uses to retrieve progress-aligned context for action generation; the chart compares progress rates on new tasks for π0.5, RICL and VICTR.

From Figure 2: VICTR method overview. The IC-VFE conditions on the reference demonstration, the shortest in the pool,... — From Figure 2: VICTR method overview. The IC-VFE conditions on the reference demonstration, the shortest in the pool, to estimate progress for the current observation. This estimate, optionally fused with visual similarity, retrieves a context chunk from the demonstration pool. The IC-VLA conditions on that chunk to predict the next action chunk, with no parameter updates.

From Figure 4: Success and task progress rates for π0.5 and three retrieval-based variants on in-domain (InD) and out... — From Figure 4: Success and task progress rates for π0.5 and three retrieval-based variants on in-domain (InD) and out-of-domain (OOD) real-world tasks. The solid segment of each bar is binary success; the lighter segment adds partial credit for annotated milestones.

Abstract

Recent advances in robot foundation models have highlighted the potential of in-context learning (ICL) for adapting robot behavior from demonstrations without further fine-tuning. Previous work has used retrieval-based post-training to equip pretrained vision-language-action (VLA) models with the capability for ICL by teaching them to condition action predictions on retrieved visually-relevant demonstration chunks. However, the effectiveness of retrieval hinges on selecting context relevant to the current execution stage, which visual similarity alone may not capture. We introduce VICTR, a framework enabling in-context learning in VLAs via progress-informed context retrieval built around an In-Context Value Function Estimator (IC-VFE). IC-VFE conditions on a reference demonstration video, a language instruction and the current observation to estimate task-relative progress. By conditioning progress estimation on a reference demonstration, IC-VFE can align the current state with a demonstrated execution even for tasks unseen during training. VICTR uses the estimated progress to select demonstration chunks at corresponding stages of execution and supplies them to a $π_{0.5}$ policy post-trained for in-context action generation. On unseen out-of-domain real-world manipulation tasks, VICTR raises the success rate from 6.2% for a fine-tuned $π_{0.5}$ baseline to 23.8% and more than doubles the task progress rate, from 28.9% to 55.5%. Project page with code, data and videos: https://ai4ce.github.io/VICTR/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong