Old Experience, New Lessons: Relabeling Autonomous Robot Rollouts in Interactive Imitation Learning

TL;DR In interactive imitation learning, SeRA has later policy checkpoints relabel the robot's own unlabeled rollouts, keeps only the relabels the checkpoints agree on, and trains a diffusion policy toward sets of actions around them. On a real tomato-picking task it reaches 80% success against 40% for the SDP baseline, with the same human feedback.
What they do
In interactive imitation learning, a human corrects the robot during deployment. This yields a few corrections and a lot of autonomous robot experience without action labels. Earlier methods (IWR, CEILing, Sirius) imitate those historical robot actions, which can reinforce suboptimal behavior. SeRA instead relabels the experience with newer policies and supervises the policy with tolerant, set-valued targets. It builds on Set-Supervised Diffusion Policy (SDP).
How it works
- Cross-round relabeling. At each unlabeled state, all later checkpoints are queried and the latest one supplies the relabel. A step is accepted only if at least 10 later checkpoints agree (low variance) on it and on every earlier step of the chunk.
- Set-valued targets. Each accepted relabel defines a ball of desired actions around it, with radius equal to its distance from the historical robot action. Human corrections get narrower balls.
- Step-masked denoising loss. Chunks that mix human actions and relabels are kept. Steps after the first failed agreement test are masked out of the diffusion loss instead of discarding the chunk.
Results
- Five simulated tasks (PushT, PickCan, Square, TwoArmLift, ThreePiece), online and offline, with a simulated teacher: SeRA matches or beats every baseline on every task, e.g. 79.8% vs 57.9% for SDP on online PushT. Every baseline that learns from the autonomous experience falls below SDP on at least four of five tasks in each setting.
- Offline ablations: without agreement filtering, Square drops from 96.3% to 31.4% and ThreePiece from 98.9% to 9.5%. Near point-valued targets drop them to 59.4% and 36.3%.
- Real robot, picking two tomatoes into a bowl (20 trials per method, 14 h training each): SeRA reaches 80%, the StepMask variant without relabeling 60%, and SDP 40%. IWR does worst and fails every trial with the top tomato at the back.
Limitations
Assumes the desired set around each accepted relabel contains an optimal action, which checkpoint agreement cannot guarantee because later policies can share errors. The relabel radius is fixed, and relabels that fail the agreement test are discarded.
Figures



Abstract
Interactive imitation learning enables robots to learn from human interventions during deployment. These interactions naturally produce sparse human corrections and abundant autonomous robot experience without human action labels. Yet existing methods either leave this experience unused for action supervision or imitate historical robot actions, which can reinforce suboptimal behavior. To address this limitation, we propose Set-valued Relabeling of Autonomous experience (SeRA), a framework that learns from autonomous robot experience through set-valued supervision of policy relabels. Later policy checkpoints relabel this experience, and agreement across checkpoints filters the relabels. Since agreement does not guarantee correctness, SeRA constructs desired action sets around accepted relabels. Following Set-Supervised Diffusion Policy (SDP), it trains an action-chunking diffusion policy to increase the probability of actions within these sets. A step-masked denoising loss combines accepted relabels with available human corrections in partially supervised chunks. Experiments across five simulated manipulation tasks in online and offline settings show that SeRA improves policy performance over SDP and outperforms baselines that imitate historical robot actions or exact relabels. On a real-robot tomato-picking task, SeRA achieves 80% success, compared with 40% for SDP, using the same human feedback and training budget. These results demonstrate that relabeled autonomous robot experience can complement human corrections to improve policy performance.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
