← Paper Digest

Old Experience, New Lessons: Relabeling Autonomous Robot Rollouts in Interactive Imitation Learning

Zhaoting Li, Cosimo Della Santina, Jens Kober

November 2026

TL;DR In interactive imitation learning, SeRA has later policy checkpoints relabel the robot's own unlabeled rollouts, keeps only the relabels the checkpoints agree on, and trains a diffusion policy toward sets of actions around them. On a real tomato-picking task it reaches 80% success against 40% for the SDP baseline, with the same human feedback.

What they do

In interactive imitation learning, a human corrects the robot during deployment. This yields a few corrections and a lot of autonomous robot experience without action labels. Earlier methods (IWR, CEILing, Sirius) imitate those historical robot actions, which can reinforce suboptimal behavior. SeRA instead relabels the experience with newer policies and supervises the policy with tolerant, set-valued targets. It builds on Set-Supervised Diffusion Policy (SDP).

How it works

  • Cross-round relabeling. At each unlabeled state, all later checkpoints are queried and the latest one supplies the relabel. A step is accepted only if at least 10 later checkpoints agree (low variance) on it and on every earlier step of the chunk.
  • Set-valued targets. Each accepted relabel defines a ball of desired actions around it, with radius equal to its distance from the historical robot action. Human corrections get narrower balls.
  • Step-masked denoising loss. Chunks that mix human actions and relabels are kept. Steps after the first failed agreement test are masked out of the diffusion loss instead of discarding the chunk.

Results

  • Five simulated tasks (PushT, PickCan, Square, TwoArmLift, ThreePiece), online and offline, with a simulated teacher: SeRA matches or beats every baseline on every task, e.g. 79.8% vs 57.9% for SDP on online PushT. Every baseline that learns from the autonomous experience falls below SDP on at least four of five tasks in each setting.
  • Offline ablations: without agreement filtering, Square drops from 96.3% to 31.4% and ThreePiece from 98.9% to 9.5%. Near point-valued targets drop them to 59.4% and 36.3%.
  • Real robot, picking two tomatoes into a bowl (20 trials per method, 14 h training each): SeRA reaches 80%, the StepMask variant without relabeling 60%, and SDP 40%. IWR does worst and fails every trial with the top tomato at the back.

Limitations

Assumes the desired set around each accepted relabel contains an optimal action, which checkpoint agreement cannot guarantee because later policies can share errors. The relabel radius is fixed, and relabels that fail the agreement test are discarded.

Figures

From Figure 1: SeRA framework. Top: during online interaction, human corrections go to a correction dataset and auton...
From Figure 1: SeRA framework. Top: during online interaction, human corrections go to a correction dataset and autonomous robot experience to an unlabeled dataset. Bottom: a later policy relabels the unlabeled observations with new action chunks (green) in place of the historical robot actions (orange); accepted relabels give desired action sets for training.
From Figure 2: Later policy checkpoints relabel missing teacher actions, and their agreement sets the acceptance mask...
From Figure 2: Later policy checkpoints relabel missing teacher actions, and their agreement sets the acceptance mask. Unlabeled chunks use relabels up to the first disagreement (top); partially labeled chunks combine the initial teacher actions with accepted relabels (bottom).
From Figure 4: Real-robot tomato picking. Top: the two configurations, with the top tomato at the front (left) or at ...
From Figure 4: Real-robot tomato picking. Top: the two configurations, with the top tomato at the front (left) or at the back (right). Bottom: successful trials for each method, out of 10 per configuration.
Abstract

Interactive imitation learning enables robots to learn from human interventions during deployment. These interactions naturally produce sparse human corrections and abundant autonomous robot experience without human action labels. Yet existing methods either leave this experience unused for action supervision or imitate historical robot actions, which can reinforce suboptimal behavior. To address this limitation, we propose Set-valued Relabeling of Autonomous experience (SeRA), a framework that learns from autonomous robot experience through set-valued supervision of policy relabels. Later policy checkpoints relabel this experience, and agreement across checkpoints filters the relabels. Since agreement does not guarantee correctness, SeRA constructs desired action sets around accepted relabels. Following Set-Supervised Diffusion Policy (SDP), it trains an action-chunking diffusion policy to increase the probability of actions within these sets. A step-masked denoising loss combines accepted relabels with available human corrections in partially supervised chunks. Experiments across five simulated manipulation tasks in online and offline settings show that SeRA improves policy performance over SDP and outperforms baselines that imitate historical robot actions or exact relabels. On a real-robot tomato-picking task, SeRA achieves 80% success, compared with 40% for SDP, using the same human feedback and training budget. These results demonstrate that relabeled autonomous robot experience can complement human corrections to improve policy performance.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.