Imitation learning and data collection
Learning from demonstrations, from teleoperation and handheld rigs to human video.
Start here: seminal papers
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning★ Must read
November 2010
DAgger trains an imitation policy by running the current policy, having the expert label the states it visits, and retraining on all data collected so far. In Super Tux Kart its steering policy never falls off the track after 15 iterations, while SMILe still falls about twice per lap.
Learning Dexterous Manipulation for a Soft Robotic Hand from Human DemonstrationSeminal
March 2016
Teaches the soft, pneumatic RBO Hand 2 from demonstrations that record only how a human moved the object. RL learns which demonstrations each controller can actually follow, and guided policy search merges the controllers into one neural network policy that turns a valve and pushes abacus beads from several hand positions, close to an oracle given the right demonstrations, and grasps a bottle in 10 of 10 trials.
Generative Adversarial Imitation LearningSeminal
June 2016
Learn a policy straight from expert demonstrations by training it against a discriminator that tells its state-action pairs from the expert's, with no reward and no IRL inner loop. On the MuJoCo tasks it almost always reaches at least 70% of expert performance, and on Humanoid it matches the expert, where behavioral cloning stays under 60%.
One-Shot Imitation LearningSeminal
March 2017
One network is trained on pairs of demonstrations across 140 block-stacking tasks so that a single demonstration of a new task turns it into a policy for that task. With soft attention over the demonstration and the blocks, it succeeds 50.6% of the time on unseen 6-stage tasks, against 49.4% for the scripted demonstrator.
Time-Contrastive Networks: Self-Supervised Learning from VideoSeminal
April 2017
Learn an image embedding from unlabeled multi-view video by pulling together simultaneous frames from different cameras and pushing apart nearby frames in time. Used as a reward, it lets a real KUKA arm learn to pour beads from a single video of a human pouring in 10 RL iterations.
One-Shot Visual Imitation Learning via Meta-LearningSeminal
September 2017
Meta-train a vision-based policy with MAML so that one gradient step on a single demonstration adapts it to a new task. A real PR2 places held items onto unseen containers with 90% one-shot success, against 25% for LSTM and contextual baselines.
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality TeleoperationSeminal
October 2017
A consumer VR headset and hand controllers are used to teleoperate a PR2, and behavioral cloning on the recorded demos learns pixel-to-action policies for ten real manipulation tasks. With under 30 minutes of demonstrations per task, success rates range from 80% to 98.9%.
HG-DAgger: Interactive Imitation Learning with Human ExpertsSeminal
October 2018
A DAgger variant where the human expert decides when to take over from the learner and keeps full control until handing back, with labels recorded only during those takeovers. On a real car weaving between obstacles, the HG-DAgger policy had no collisions or road departures, against 1 collision and 6 departures for behavioral cloning.
RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through ImitationSeminal
November 2018
A crowdsourcing platform where remote workers teleoperate a simulated robot arm in 6-DoF by moving an iPhone while watching a video stream in their browser. In 22 hours of use it collected over 2200 successful demonstrations, and using 1000 of them raises sparse-reward RL return on can picking from 0 to 641 out of 1000.
Learning Latent Plans from PlaySeminal
March 2019
Play-LMP learns one goal-conditioned policy from unlabeled, teleoperated play by encoding random windows of play into a latent plan space. On 18 simulated manipulation tasks it reaches 85.5% success from state observations, against 70.3% for 18 task-specific behavioral cloning policies.
RLBench: The Robot Learning Benchmark & Learning EnvironmentSeminal
September 2019
RLBench is a simulated benchmark of 100 hand-designed manipulation tasks for a Franka Panda, each with unlimited motion-planned demonstrations and text descriptions. The paper proposes a few-shot challenge on 10 held-out tasks but reports no baseline results.
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping from Low-Cost DemonstrationsSeminal
December 2019
People record 12 hours of grasping demonstrations in everyday places with a $600 handheld grabber that carries a wrist RGB-D camera, and these pretrain a closed-loop 6DoF grasping Q-function that scores rendered future views. After finetuning on a UR5, it grasps novel objects 92% of the time in static scenes and 88% while the objects are being moved.
robosuite: A Modular Simulation Framework and Benchmark for Robot LearningSeminal
September 2020
A modular MuJoCo framework that builds robot-learning tasks from interchangeable robots, grippers, controllers and objects, with nine standardized manipulation benchmarks. Soft Actor-Critic baselines solve three of the nine tasks and learn faster with operational-space control than with joint-velocity control.
Transporter Networks: Rearranging the Visual World for Robotic ManipulationSeminal
October 2020
Transporter Networks pick a spot, then find where it goes by matching a rotated crop of deep features around the pick against the whole top-down scene. With 100 demonstrations they reach over 90% success on 6 of 10 simulated tasks, and 98.9% on real kit assembly.
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation★ Must read
August 2021
A controlled study of six offline learning algorithms on human teleoperated demonstrations for five simulated and three real manipulation tasks. History-aware BC-RNN wins, reaching 71.3% on two-arm Transport versus 17.3% for plain BC, while offline RL methods BCQ and CQL struggle on human data.
DexMV: Imitation Learning for Dexterous Manipulation from Human VideosSeminal
August 2021
DexMV records human hand videos, estimates 3D hand and object poses, and retargets them into demonstrations for a 30-DoF simulated Adroit hand. With these demonstrations DAPG relocates four of five YCB objects at 93-100% success, where RL alone reaches 0-67%.
Implicit Behavioral CloningSeminal
September 2021
Behavioral cloning with an energy-based model, where the policy outputs the action that minimizes a learned energy of observation and action. On four real-robot pushing tasks from images it beats an otherwise identical MSE policy, with 83% vs 7% success on a 1 mm-tolerance insertion.
CLIPort: What and Where Pathways for Robotic ManipulationSeminal
September 2021
CLIPort adds a frozen CLIP stream to the Transporter pick-and-place network, so one imitation-learned policy follows language instructions on tabletop tasks. In simulation it is the only model to average above 90% on seen tasks, and a single real-robot policy learns 9 tasks from 179 image-action pairs with 55-75% success.
CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation TasksSeminal
December 2021
CALVIN is an open-source simulated benchmark in which a Franka arm must follow chains of five natural-language instructions across four related tabletop environments, learning from 24 hours of unlabeled teleoperated play. A multi-context imitation learning baseline solves 53.9% of single tasks but completes all five instructions in a row in only 0.08% of chains.
BC-Z: Zero-Shot Task Generalization with Robotic Imitation LearningSeminal
February 2022
BC-Z trains one behavior-cloning policy on 25,877 robot demonstrations of 100 manipulation tasks, conditioned on a sentence or a human video. Given only a new sentence, it succeeds at least sometimes on 24 of 29 unseen tasks, averaging 44% success on those 24.
Behavior Transformers: Cloning $k$ modes with one stoneSeminal
June 2022
Behavior Transformer (BeT) clones unlabeled, multi-modal demonstrations by having a small transformer pick a k-means action bin and then add a learned continuous offset. On multimodal block pushing it gets both blocks into their squares in 71% of rollouts, where every baseline gets 0%, while still covering the demonstrated modes.
Perceiver-Actor: A Multi-Task Transformer for Robotic ManipulationSeminal
September 2022
PerAct encodes a language goal and a voxelized RGB-D scene with a Perceiver Transformer and picks the next gripper pose by classifying voxels. One agent trained on 18 RLBench tasks beats a 3D ConvNet baseline by 2.83x with 100 demos per task, and a real Franka learns 7 tasks from 53 demos.
Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During DeploymentSeminal
November 2022
Sirius deploys a learned manipulation policy with a human who takes over when it is about to fail, then retrains the policy on that deployment data, upweighting interventions and discarding the robot steps just before them. Autonomous success beats state-of-the-art baselines by 8% in simulation and 27% on a real robot.
See, Hear, and Feel: Smart Sensory Fusion for Robotic ManipulationSeminal
December 2022
A robot arm with a camera, a contact microphone and GelSight fingertip sensors fuses all three senses with self-attention and learns dense packing and pouring from demonstrations. With all three it inserts a peg past hidden bumps in 100% of trials (vision alone 25%) and pours to within 1.41 g of a 40 g target.
VideoDex: Learning Dexterity from Internet VideosSeminal
December 2022
Retarget human hand motion from Epic Kitchens videos into robot hand and arm trajectories, pretrain an open-loop policy on them, then finetune on teleoperated demos. On a LEAP Hand and xArm, VideoDex picks unseen objects with 0.77 success versus at most 0.44 for baselines without the human-video prior.
MimicPlay: Long-Horizon Imitation Learning by Watching Human PlaySeminal
February 2023
MimicPlay learns a goal-conditioned latent planner from 10 minutes of human hand play video per scene, then trains a low-level robot policy on its plans from 20-40 teleoperated demos. On long-horizon kitchen tasks it succeeds 70% of the time with 40 demos, against at most 17% for prior imitation-learning baselines.
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion★ Must read
March 2023
Represent a visuomotor policy as a denoising diffusion process that generates a short sequence of future actions, executed with receding-horizon control. It beats prior behavior-cloning methods by 46.9% on average across simulation benchmarks and reaches 95% success on a real Push-T task.
Affordances from Human Videos as a Versatile Representation for RoboticsSeminal
April 2023
VRB learns from EPIC-Kitchens human videos where a person would touch a scene and how their hand moves afterwards, then uses these predictions to drive four kinds of robot learning. With imitation from data collected this way, real robots average 57% success across 8 tasks, against 25% for the next-best affordance model.
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware★ Must read
April 2023
ALOHA is a bimanual teleoperation rig built from off-the-shelf arms for under $20k, and ACT is an imitation learning policy that predicts chunks of future joint targets with a transformer CVAE. From 50 demonstrations per task, ACT slots a battery 96% of the time and opens a translucent condiment cup 84% of the time.
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot LearningSeminal
June 2023
A simulated benchmark for lifelong robot learning, with 130 procedurally generated manipulation tasks in four suites and 50 human demos per task. In its study, plain sequential finetuning beats the lifelong-learning methods tested at forward transfer, and supervised pretraining on 90 tasks hurts later lifelong learning (AUC 0.32 to 0.22 for ER with ResNet-T).
BridgeData V2: A Dataset for Robot Learning at ScaleSeminal
August 2023
A reusable real-robot dataset of 60,096 language-labeled trajectories (13 skills, 24 environments) collected with a WidowX 250 setup costing about $4,000. Six imitation and offline RL methods trained on it reach up to 0.60 average success on unseen objects and environments, and all keep non-zero success in a second lab.
GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators★ Must read
September 2023
Build a half-scale, kinematically equivalent copy of a robot arm from 3D-printed parts and hobby servos for under $300, and stream its joint angles straight to the real arm. In a 12-person bimanual user study GELLO averaged 0.92 task success, against 0.72 for VR controllers and 0.63 for 3D mice.
MimicGen: A Data Generation System for Scalable Robot Learning using Human DemonstrationsSeminal
October 2023
MimicGen turns about 10 human demos per task into 1000 new ones by re-posing object-centric pieces of them for new scenes and keeping only the replays that succeed. From ~200 human demos it generated 50K+ across 18 tasks; on Three Piece Assembly, agent success rises from 1.3% to 82.0%.
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body TeleoperationSeminal
January 2024
A $32k bimanual mobile manipulator whose operator, tethered to the base at the waist, drives the wheels while puppeteering both arms, plus behavior cloning co-trained on existing static ALOHA data. With 20-50 demos per task, ACT reaches 80-95% success on six of seven household tasks.
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots★ Must read
February 2024
A hand-held GoPro gripper with a fisheye lens and side mirrors records demonstrations anywhere, and a latency-matched Diffusion Policy with relative actions transfers them directly to robots. Trained on 1400 in-the-wild demos, a cup-arrangement policy succeeds 71.7% of the time in unseen environments.
3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D RepresentationsSeminal
March 2024
DP3 conditions a diffusion policy on a 64-dim feature from a small MLP encoder over a sparse, colorless point cloud from a single camera. Across 72 simulation tasks it averages 74.4% success versus 59.8% for image-based Diffusion Policy, and it reaches 85% on 4 real tasks from 40 demonstrations each.
DexCap: Scalable and Portable Mocap Data Collection System for Dexterous ManipulationSeminal
March 2024
A wearable rig of mocap gloves, SLAM wrist cameras and a chest RGB-D camera records human hand motion with 3D point clouds, and DexIL retargets it to two LEAP hands and trains a point-cloud diffusion policy. With no robot data, policies reach 72% average success on three tasks, and collection is three times faster than teleoperation.
DROID: A Large-Scale In-The-Wild Robot Manipulation DatasetSeminal
March 2024
A teleoperated manipulation dataset of 76k demonstrations across 564 scenes and 86 tasks, collected on 18 identical Franka setups at 13 institutions. Co-training a diffusion policy on DROID beats the next best training mix by 22 points of success in distribution and 17 points out of distribution.
HumanPlus: Humanoid Shadowing and Imitation from HumansSeminal
June 2024
A humanoid learns to copy a person's body and hand motion live from one RGB camera, using an RL policy trained on 40 hours of human motion data; the copied demonstrations then train vision-based skill policies. With up to 40 demonstrations, a 33-DoF humanoid completes six tasks, from folding a sweatshirt to putting on a shoe and walking, at 60-100% success.
3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile SensingSeminal
October 2024
Thin piezoresistive tactile pads (16x16 per finger) on soft grippers, with tactile readings placed as points next to the camera point cloud and fed to a diffusion policy. Across four bimanual tasks it reaches 80-90% whole-task success, against 45-65% for point-cloud-only policies.
DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation LearningSeminal
October 2024
DexMimicGen extends MimicGen to two-armed and dexterous-hand robots, turning 60 teleoperated demos into 21K simulated demos across nine tasks. On a real GR-1 humanoid, a policy trained on 40 generated demos sorts cans with 90% success, against 0% from the 4 source demos alone.
Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich ManipulationSeminal
March 2025
A slow latent diffusion policy plans action chunks at 1-2 Hz while a fast decoder adjusts them from tactile or force feedback at 24 Hz, trained on demos from an AR teleoperation system that shows touch. On peeling, wiping and bimanual lifting it beats Diffusion Policy by a wide margin (peeling score 0.95 vs 0.44).
DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous ManipulationSeminal
May 2025
DexUMI records dexterous-hand demonstrations with an exoskeleton worn on the human hand that matches the robot hand's fingertip kinematics, then paints the robot hand over the human hand in the wrist-camera video. Policies trained only on this data average 86% success on four real tasks across two robot hands, and collection is 3.2x faster than teleoperation.
Recent
OopsieRecover: Learning Transferable VLA Corrections from Semantic Interventions
November 2026
OopsieRecover turns a handful of human interventions, each with a short note on why, into gated costs over 3D point sets that decide when and how to steer a frozen pi0.5 VLA. On a real DROID setup it raises task progress from 35.1% for the next-best method to 61.0%, and the corrections carry over to new object poses and scenes.
Latent RL Priors for Data-Efficient Dexterous Manipulation
November 2026
Distill a large-scale simulated RL policy for a KUKA arm with a dexterous hand into a frozen flow-matching action decoder, then learn only a visual front end from a few demonstrations. In simulation it beats π0.5 and GR00T N1.7 fine-tuned on the full datasets using 10x fewer demos on door lifting and 5x fewer on bimanual assembly.
MOMASPLAT: Dynamic Novel-View Demonstration Augmentation for Mobile Manipulation
November 2026
MoMaSplat rebuilds the scene and the manipulated object as separate Gaussian splats from a mobile robot's own data, then re-renders whole demonstrations, including the object moving, from new base poses. On five real TIAGo tasks it succeeds on 37% of rollouts from unseen base poses, against 17% for the best baseline.
Spatial-UMI-Dream: Scaling World Action Models with Metric Geometry for UMI Manipulation
November 2026
Spatial-UMI-Dream pretrains a 5B world action model on 605 hours of UMI handheld-gripper data to predict future RGB, stereo-derived metric depth and actions together. It reaches 85.0% average success on six real bimanual tasks, 24.2 points above the best baseline (π0.5).
Which Physical Tactile Cues Matter for Contact-Rich Imitation Learning? A Controlled Real-Robot Study
November 2026
From one GelSight-style sensor stream they derive tactile RGB, normal/depth maps and marker flow, and compare them as Diffusion Policy inputs over 720 real-robot trials. Touch raises success from 57.8% to 78.9%, and marker flow is the best single cue at 74.4%.
Online Adaptation to Latent Physical Properties for Learning from Demonstration
November 2026
A Diffusion Policy is conditioned on a latent physical-property estimate that is updated online from a learned dynamics model's prediction errors. This lets the robot regrasp or re-push objects that look identical but have different centers of mass. On a real xArm7 it raises pouch-placement success from 40.7% to 66.7% over a standard Diffusion Policy.
Correct Robots Before They Make Mistakes: Proactive Human-in-the-Loop Intervention via Preference Learning
November 2026
An XR headset shows the operator the policy's predicted gripper path so they can take over before a mistake, and each rejected prediction is paired with the correction that followed for preference post-training. With five correction episodes, mean real-robot success over HG-DAgger rises from 44.0% to 70.3% for a diffusion policy and from 57.8% to 76.9% for X-VLA.
ELA: End-to-End Latent Action Policy Learning from Video Demonstrations
November 2026
ELA trains a latent action model, a latent-action policy and a small action decoder together, so an imitation policy can learn from human or unlabeled robot video plus a few action-labeled demos. With 10 robot demos and 40 human videos per task, it raises real-robot success from 48% (Diffusion Policy) to 70%.
GUILD: Guided Iterative Data Collection for Imitation Learning
November 2026
GUILD tells teleoperators which tasks to collect next. A VLM proposes candidate tasks in accessible environments, and data attribution scores how much each task's demos help a hard-to-reach deployment task. In open-ended real-world collection it beats uniform collection over the same VLM proposals by about 25% average success, and beats two expert roboticists on average.
RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On
November 2026
RoboTryOn turns egocentric human videos into robot demonstrations by generating a video of the robot making the same motion and reading its actions back with an inverse dynamics model. On two bimanual robots with dexterous hands, it lifts success on seven tasks that have no robot demonstrations from 2% to 60% on average.
Coordinating Humanoid Loco-Manipulation through Action-Conditioned Latent Foresight
November 2026
A humanoid policy samples several whole-body and hand command sequences, predicts in latent space where each would lead, and executes the one that makes the most visual progress and is most temporally consistent. On a Unitree G1 it succeeds 85.7% on seven tasks (65.7% with unseen objects) versus 70.0% (48.6%) for GR00T N1.7.
AIRoA-0: Long-Horizon, Precise, Cross-Embodiment Manipulation from Handheld Data
November 2026
Pretrain a π0.5-based policy on 16,151 hours of handheld-gripper demonstrations with no robot data at all, then post-train it per task on a few hours of handheld data. On ten long bimanual tasks at a site outside its data, A-0 averages 78% success against 27% for the best of four post-trained VLA baselines.
Robot-Matched Soft Fingers for UMI Data Collection: Lessons from 100 Hours of Pick and Place
November 2026
A battery-powered handheld UMI with soft, angled fingers that match the robot's gripper, used to collect 99.7 hours of pick and place over 53 objects. Mid-training π0.5 on this data raises success after fine-tuning on unseen objects from 35% to 57% on a Galaxea R1 Pro.
Pixel: A Co-Designed Full-Stack Robot System for Learning General-Purpose Mobile Manipulation
November 2026
Pixel is a wheeled humanoid that costs under $15,000. The robot, its teleoperation interfaces, a wearable human-demonstration rig, the whole-body controller and the policy were designed together, so human and robot data share one action space. Policies trained only on wearable-rig demonstrations reach 82.0% average task progress on three household tasks, and collecting those demonstrations is 4.2-6.2x faster than teleoperation.
Pixel-UMI: Learning Environment-Aware Whole-Body Manipulation from Sensorized Human Demonstrations
November 2026
Pixel-UMI turns head-and-gripper human demonstrations into whole-body robot training data by retargeting them against a 3D reconstruction of each scene, so the policy learns postures that clear obstacles. Across five real mobile-manipulation tasks it reaches 87.0% average task progress, and scene-aware retargeting alone adds 64 points.
PanoTac: A Dense Three-Axis Tactile Sensor for Robust Touch-Centric Manipulation
November 2026
A sub-$50 gripper fingertip that fuses a dense piezoresistive array with magnets and magnetometers into a 12x32 map of normal and shear force. With it, visuo-tactile diffusion policies reach 91.7% success on test-tube prep, grain scooping and salt pouring, against 45% for vision alone.
ReContact: Reconstruct and Retarget Human Contacts from Video for Contact-Rich Humanoid Manipulation
November 2026
ReContact turns a human video into an object-centric contact plan (which body part touches which point of the object, pressing in which direction) and trains a humanoid policy rewarded for following it. In simulation it completes a thigh-supported box regrasp into a two-palm bottom hold in 83.5% of episodes versus at most 0.16% for adapted baselines, and it runs nine tasks on a real Unitree G1.
Visuo-tactile Diffusion Policy for Robotic Manipulation with Auxiliary Tactile Dynamics Prediction
November 2026
A visuotactile diffusion policy for simulated UR5 assembly, with a training-only head that predicts how the fingertip images change two steps ahead. The full model reaches 59.0% success against 42.7% for vision only, while adding touch without the auxiliary loss drops to 33.7%.
DexPort: A Hand-Held Skill Interface for Dexterous Hand Data Collection
October 2026
A hand-held rig that carries the robot's own 20-DoF hand and camera, so demonstrations need no retargeting. Simple joystick skills drive the fingers, and the commanded joint targets, not measured states, are logged as actions; that alone lifts policy success from 11% to 64%.
New papers stay for 14 days.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.