Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning

Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Fan, Yuke Zhu

Seminal October 2024

Pipeline: human teleoperation, 5 source demos, 1000 generated demos.

TL;DR DexMimicGen extends MimicGen to two-armed and dexterous-hand robots, turning 60 teleoperated demos into 21K simulated demos across nine tasks. On a real GR-1 humanoid, a policy trained on 40 generated demos sorts cans with 90% success, against 0% from the 4 source demos alone.

What they do

Teleoperating two arms with multi-fingered hands is slow, so human demos are scarce. DexMimicGen generates thousands more in simulation from a handful. It extends MimicGen, built for one arm with a parallel-jaw gripper, to two arms that work independently, together or in a fixed order.

How it works

  • Per-arm subtasks. Each demo is segmented into object-centric subtasks separately for each arm. Each arm has its own action queue, refilled with the next source segment transformed to the current object pose, so the arms need not finish at the same time.
  • Coordination subtasks (e.g. lifting a tray together) end at the same timestep in the source demo. Both arms use one shared transform and wait for each other to stay in sync. Handovers instead replay the source trajectory untransformed to stay within kinematic limits.
  • Sequential subtasks carry an ordering constraint: one arm waits until the other finishes its pre-subtask. Finger joint actions are replayed from the source demo. Only successful generated demos are kept.

Results

  • 21K demos across nine tasks and three embodiments from 60 source demos (5 per dexterous task). With Diffusion Policy, success rises from source demos alone to 1000 generated demos: Piece Assembly 3.3% to 80.7%, Threading 1.3% to 69.3%, Drawer Cleanup 0.7% to 76.0%.
  • Replaying source demos with action noise instead gives 12.7-26.7% success, against 74.0-79.3% for DexMimicGen on the same tasks.
  • Going from 100 to 1000 demos helps a lot; 5000 does not always beat 1000.
  • Real GR-1 can sorting: 4 teleoperated demos produce 40 demos generated in a digital twin and executed on the real robot. Over 20 trials, the policy succeeds 90%, against 0% from the source demos.

Limitations

Collisions are not handled, and some generated demos fail on them. Subtasks are segmented and labelled by hand or by heuristics, and object poses must be known before contact. Finger actions are replayed, so they vary only locally. The real-world pipeline needs a digital twin and manual resets.

Videos

Generated Pouring demo: the humanoid pours a ball into a bowl, then moves the bowl to the pad.
Generated Tray Lift demo with two arms lifting the tray together.

Figures

From Figure 1: A human teleoperates about five demos, which are replayed in a digital twin. DexMimicGen generates man...
From Figure 1: A human teleoperates about five demos, which are replayed in a digital twin. DexMimicGen generates many more trajectories in simulation, and a policy trained on them is deployed on the real humanoid.
From Figure 3: Left, each source demo is segmented per arm, and the reference object pose of each segment is recorded...
From Figure 3: Left, each source demo is segmented per arm, and the reference object pose of each segment is recorded. Right, in a new scene, source segments are transformed by the reference object poses and executed.
From Figure 4: The nine simulation tasks, three per embodiment: two arms with parallel-jaw grippers (top), two arms w...
From Figure 4: The nine simulation tasks, three per embodiment: two arms with parallel-jaw grippers (top), two arms with dexterous hands (middle) and a humanoid (bottom).
From Figure 5: Success rates of policies trained on 100, 500, 1000 and 5000 generated demos for three tasks.
From Figure 5: Success rates of policies trained on 100, 500, 1000 and 5000 generated demos for three tasks.
Abstract

Imitation learning from human demonstrations is an effective means to teach robots manipulation skills. But data acquisition is a major bottleneck in applying this paradigm more broadly, due to the amount of cost and human effort involved. There has been significant interest in imitation learning for bimanual dexterous robots, like humanoids. Unfortunately, data collection is even more challenging here due to the challenges of simultaneously controlling multiple arms and multi-fingered hands. Automated data generation in simulation is a compelling, scalable alternative to fuel this need for data. To this end, we introduce DexMimicGen, a large-scale automated data generation system that synthesizes trajectories from a handful of human demonstrations for humanoid robots with dexterous hands. We present a collection of simulation environments in the setting of bimanual dexterous manipulation, spanning a range of manipulation behaviors and different requirements for coordination among the two arms. We generate 21K demos across these tasks from just 60 source human demos and study the effect of several data generation and policy learning decisions on agent performance. Finally, we present a real-to-sim-to-real pipeline and deploy it on a real-world humanoid can sorting task. Generated datasets, simulation environments and additional results are at https://dexmimicgen.github.io/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.