← Paper Digest

Hindsight Self-Distillation: Teaching Robots to Act from the Present

Haisen He, Yechi Liu, Tianle Shi, Haoyun Liu, Chuanzhang Meng, Tiancai Jiao, Yifan Bai, Xinyuan Chang, SongLin Dong, Zhiheng Ma

November 2026

TL;DR HSD trains a video-backbone robot policy with a teacher branch that also sees later frames of each demonstration, and distills its action predictions and visual summaries into a student that acts from current images only. It reaches 98.50% on LIBERO and 92.31% on RoboTwin 2.0, and raises real bimanual success from 24.17% to 37.50% over Fast-WAM.

What they do

A recorded demonstration shows what happened after each action, but a deployed policy only sees the present. Hindsight Self-Distillation (HSD) feeds those later frames to a teacher branch during training, and the teacher then supervises a student branch that acts from current images alone. Both branches live in one policy built on Fast-WAM, and only the student runs on the robot.

How it works

  • Shared model. A Video DiT (Wan2.2-TI2V-5B backbone) and an Action DiT generate 32-step action chunks by flow matching. Teacher and student share both transformers but have separate sets of 32 learned queries.
  • Summary-only vision. Teacher queries attend to the current frames plus eight later frames per camera; student queries attend to the current frames only. Each branch’s query states are the only visual input to its action stream.
  • Two-level distillation. For each noisy action chunk, the student is trained on either the demonstrated flow or the teacher’s detached prediction (50/50), and its final visual summaries are aligned to the teacher’s.
  • Deployment. The student encodes the current images once per chunk and reuses the cached summaries for all ten denoising steps.

Results

  • LIBERO: 98.50% average, against 96.95% for their Fast-WAM reproduction and 97.05-97.30% for Being-H0.7- and FLARE-style variants. On LIBERO-Long, 97.8% against 93.4-93.6%.
  • RoboTwin 2.0: 92.31% (92.48% clean, 92.14% randomized), 0.60 and 0.36 points above published Fast-WAM. On LIBERO-Plus variants, 63.71% against 49.43% for the released Fast-WAM checkpoint.
  • Real YAM bimanual robot, six tasks with 50 demonstrations and 20 trials each: 45/120 successes (37.50%) against 29/120 (24.17%) for Fast-WAM. Plug insertion rises from 20% to 40% and water pouring from 10% to 25%.
  • Latency is 303.0 ms per action chunk, close to Fast-WAM’s 307.5 ms.

Limitations

Hindsight only helps where the teacher’s targets are predictable from the current context, and the analysis holds at fixed weights while closed-loop gains are measured empirically. On LIBERO, HSD matches rather than exceeds the published 98.50% of Cosmos Policy and Fast-WAM-Joint. The real-robot study uses 20 trials per task, average real-world success stays below 40%, and slipper placement shows no gain (30% for both).

Figures

From Figure 1: Learn with hindsight, act from the present. The demonstration records the scene before and after the r...
From Figure 1: Learn with hindsight, act from the present. The demonstration records the scene before and after the robot moves; during training the teacher uses current and later frames to guide what the student extracts from the current images; at deployment the student acts from the current images, robot state and instruction.
From Figure 2: The training-only teacher (gray) and deployable student (blue) share the Video DiT and Action DiT weig...
From Figure 2: The training-only teacher (gray) and deployable student (blue) share the Video DiT and Action DiT weights. Teacher queries summarize current and future frames, student queries only current frames, and each branch's summaries are the sole visual input to its action stream.
From Figure 4: The six real-world tasks on the YAM robot: tube insertion, slipper placement, color sorting, pen place...
From Figure 4: The six real-world tasks on the YAM robot: tube insertion, slipper placement, color sorting, pen placement, water pouring and plug insertion.
From Figure 6: Mean inference latency per action chunk on one NVIDIA H20; HSD takes 303.0 ms, close to Fast-WAM's 307...
From Figure 6: Mean inference latency per action chunk on one NVIDIA H20; HSD takes 303.0 ms, close to Fast-WAM's 307.5 ms. Lower is better.
Abstract

A robot must choose an action before it can see the outcome. In a recorded demonstration, later camera frames show what happened after the robot moved. Hindsight Self-Distillation (HSD) uses these later frames to teach a policy what to extract from current images. During training, a teacher sees both the current and later frames, while a student sees the current frames. They share a visual encoder and an action predictor. Each branch summarizes its images with a small set of feature vectors. These visual summaries supply all visual input to the action predictor. The teacher learns from demonstrated actions. For each noisy action sequence, the student learns from either the demonstration or the teacher's denoising prediction. It also learns to match the teacher's visual summaries. We analyze which aspects of the teacher's predictions the student can recover from its inputs. At deployment, the student encodes the current images once and reuses its summaries at every denoising step. HSD achieves average success rates of 98.50\% on LIBERO and 92.31\% on RoboTwin 2.0.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.