Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations

Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, Dieter Fox

Seminal October 2023

Highlights: demos with one mug become generated demos with 12 mugs, then other tasks.

TL;DR MimicGen turns about 10 human demos per task into 1000 new ones by re-posing object-centric pieces of them for new scenes and keeping only the replays that succeed. From ~200 human demos it generated 50K+ across 18 tasks; on Three Piece Assembly, agent success rises from 1.3% to 82.0%.

What they do

Imitation learning needs many teleoperated demonstrations, yet much of that data repeats the same skill in a different place. MimicGen takes a handful of human demos (10 per task) and automatically generates large datasets for wider object placements, new object instances of the same category and different robot arms. BC-RNN policies are then trained on the generated data.

How it works

  • Object-centric segments. Each task is a known sequence of subtasks, each relative to one object (grasp mug, place it on the machine, …). Source demos are split where each subtask ends.
  • Transform and stitch. For a new scene, pick a source segment, read its delta-pose actions as end-effector target poses and transform them so the pose relative to the object’s new pose stays the same. A linear interpolation segment bridges from the robot’s current pose to the start of the segment.
  • Execute and filter. The robot executes the new trajectory; only attempts that complete the task are kept.

Results

  • 50K+ demos for 18 tasks in robosuite and Isaac Gym Factory from ~200 human demos. Training on 1000 generated D0 demos instead of the 10 source demos: Square 11.3% to 90.7%, Threading 19.3% to 98.0%. Agents on the broader D1 distributions reach 42-99%.
  • 200 MimicGen demos (from 10 human) give agents comparable to 200 human demos. Going from 200 to 1000 generated demos helps a lot; 5000 adds little.
  • Panda demos generate data for Sawyer, IIWA and UR5e: generation rates differ (38-74% on Square D0) but policies reach 80-91% on all four arms. Factory assembly policies reach 82-99% on D0 vs 9-15% from the source demos.
  • Real robot: generation succeeds 82.3% (Stack) and 52.1% (Coffee); trained agents score 36% and 14% over 50 trials on the wide distribution, vs 0% for agents trained on the 10 source demos.

Limitations

Needs the subtask sequence, one reference object per subtask and object poses during generation (not at deployment). Interpolation is not collision-aware; the authors suggest the longer interpolation used on hardware (50 steps vs 5) may explain the lower real-world success. Filtering only on task success can leave biased data. Shown only on quasi-static, single-arm tasks with rigid objects from one category.

Videos

Mosaic of generated demonstrations across many tasks and robot arms.
Source human demos for Three Piece Assembly, where the base never moves.

Figures

From Figure 1: a small set of human demonstrations is re-purposed into a large dataset that spans diverse scene confi...
From Figure 1: a small set of human demonstrations is re-purposed into a large dataset that spans diverse scene configurations, objects and robot hardware.
From Figure 2: source demos are parsed into one segment per object-centric subtask (left); for a new scene, each subt...
From Figure 2: source demos are parsed into one segment per object-centric subtask (left); for a new scene, each subtask's segment is chosen from a source demo (blue border), transformed for the new scene and executed (right).
From Figure 4: success rates of image-based agents trained on 200 human demos versus 200, 1000 and 5000 MimicGen demo...
From Figure 4: success rates of image-based agents trained on 200 human demos versus 200, 1000 and 5000 MimicGen demos generated from only 10 human demos.
Abstract

Imitation learning from a large set of human demonstrations has proved to be an effective paradigm for building capable robot agents. However, the demonstrations can be extremely costly and time-consuming to collect. We introduce MimicGen, a system for automatically synthesizing large-scale, rich datasets from only a small number of human demonstrations by adapting them to new contexts. We use MimicGen to generate over 50K demonstrations across 18 tasks with diverse scene configurations, object instances, and robot arms from just ~200 human demonstrations. We show that robot agents can be effectively trained on this generated dataset by imitation learning to achieve strong performance in long-horizon and high-precision tasks, such as multi-part assembly and coffee preparation, across broad initial state distributions. We further demonstrate that the effectiveness and utility of MimicGen data compare favorably to collecting additional human demonstrations, making it a powerful and economical approach towards scaling up robot learning. Datasets, simulation environments, videos, and more at https://mimicgen.github.io .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.