Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

GUILD: Guided Iterative Data Collection for Imitation Learning

Shivin Dass, Kevin Nguyen, Rohan Yelandur, Junhong Xu, Roberto Martín-Martín

November 2026

Real-robot rollout of pink-cup-sink: pick up the pink cup and place it in the sink.

TL;DR GUILD tells teleoperators which tasks to collect next. A VLM proposes candidate tasks in accessible environments, and data attribution scores how much each task's demos help a hard-to-reach deployment task. In open-ended real-world collection it beats uniform collection over the same VLM proposals by about 25% average success, and beats two expert roboticists on average.

What they do

Robots are often deployed where collecting demonstrations is hard (an end-user home), while teleoperation is easy in other labs or mock setups. GUILD treats data collection itself as the optimization problem: given a few target demos and a fixed budget, it decides which tasks to collect in the accessible environments, instead of collecting broadly and filtering afterwards.

How it works

  • Task proposal. GPT-5.2 sees a video and description of the target task plus the available scenes and objects, and proposes tasks: scene, object configuration and a language instruction. Every few rounds it proposes more, given a utility-ranked list of earlier tasks.
  • Data evaluation. An Octo-Small policy is co-trained 50/50 with the target demos. Metagradients (MAGIC, REPLAY) give each trajectory’s effect on validation loss over the target demos; scores are aggregated into a per-task marginal utility.
  • Iterative collection. Each round, a teleoperator collects a small batch (6 demos in the real world) from the highest-utility task, then utilities are recomputed. Real-world runs use 20 rounds.

Results

  • LIBERO (3 target tasks, data from 16 other scenes): GUILD matches or exceeds the strongest baseline at every budget, reaching 64.9% average success. Eight roboticists given the same budget average below GUILD.
  • Real world (6 rooms, 4 target tasks, 20 rollouts each): selecting from a fixed 49-task dataset, GUILD beats all baselines and an All-Data policy trained on everything, including target-room data.
  • Open-ended real-world collection: about 25% higher average success than VLM-Uniform, and higher on average than two experts with 4-5 years of experience.
  • One attribution step takes 10-15 min on four A5000 GPUs; a full real-world run takes 10-15 h.

Limitations

The VLM can propose infeasible or unoriginal tasks, and swapping it changes per-task results. Smaller batches give better utility estimates but need more attribution passes. Tasks are discretized by the VLM proposals, and a few target-domain demos are still required.

Videos

Real-robot rollout of towel-drawer-close: put the towel in the open drawer, then close it.
Real-robot rollout of bowl-pour: grasp the black bowl and pour its contents into the drawer.

Figures

From Figure 1: Overview of GUILD. A VLM proposes scenes, object placements and tasks in the accessible data collectio...
From Figure 1: Overview of GUILD. A VLM proposes scenes, object placements and tasks in the accessible data collection environments; collected trajectories are scored with predictive data attribution for their effect on the deployment task, and the scores guide the next round of collection and task proposals.
From Figure 3: Real-world guided data collection. Success of policies trained on data from each collection strategy u...
From Figure 3: Real-world guided data collection. Success of policies trained on data from each collection strategy under the same budget; GUILD is highest.
From Figure 4: Average success on 3 LIBERO tasks for datasets collected by each method and by human roboticists.
From Figure 4: Average success on 3 LIBERO tasks for datasets collected by each method and by human roboticists.
Abstract

Imitation learning policies in robotics are trained on datasets collected through costly real-world teleoperation. Often, the data-collection process is unguided, relying on the teleoperator's intuition, resulting in data that contributes little to overall policy performance. Recent work has shown that curating such unguided datasets can substantially improve policy performance, but these approaches operate by filtering existing datasets after collection and therefore do not prevent the acquisition of uninformative data. In this work, we propose GUILD, a novel Guided Data Collection (GDC) method that actively reasons about which data should be collected to maximize downstream policy performance. We introduce an iterative framework that alternates between 1) task proposal and data collection and 2) data evaluation to close the loop. For task proposal, we leverage semantic priors from vision-language models to identify relevant environments, object configurations, and goals for data collection. However, since semantic relevance does not imply policy improvement, we perform data evaluation. By leveraging predictive data attribution we approximate how the collected trajectories influence policy performance on a target deployment setting. This signal is then used as feedback to guide subsequent data collection, closing the data collection loop. We evaluate our approach in simulation and real robot settings, demonstrating that guided data collection substantially improves sample efficiency and deployment performance compared to passive data collection strategies, multiple baselines and experienced roboticists. More information at https://robin-lab.cs.utexas.edu/guild2026/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.