Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning

Huaijin Pi, Zehong Shen, Yong Wang, Kechun Xu, Changpeng Wang, Dingkun Wei, Xueyang Wang, Wenjia Wang, Xing Zhu, Yujun Shen, Taku Komura

November 2026

TL;DR DexSeed turns 19 human hand-object sequences into 25,270 simulated dexterous pick-and-place demonstrations, with no teleoperation. A VLA fine-tuned only on this data reaches 91-98% success in simulation and 20-80% per grasp mode on a real Franka arm with a Wuji hand.

What they do

Teleoperating a dexterous hand to collect VLA training data is costly. DexSeed starts from human hand-object recordings (GRAB) instead and expands each one into many robot pick-and-place demonstrations that keep the original grasp. A VLA fine-tuned on the result controls a 7-DoF Franka arm and a 20-DoF Wuji hand, and the language instruction says which grasp to use.

How it works

  • Grasp bank. Sequences are clustered by a 170-dimensional grasp descriptor. Eight grasp modes are picked by hand, and each gets one GPT-written description shared by all its demonstrations.
  • Retargeting. An InteractionMesh over hand landmarks and object surface points is preserved while optimizing the robot wrist and fingers, which also transfers the grasp to resized or different object shapes.
  • Physics realization. Arm trajectories are augmented over start, object and placement poses. In IsaacLab an RL policy outputs residual finger actions while the arm tracks the precomputed joint path, so the fingers learn to correct for arm tracking error. Only successful rollouts are kept.
  • Rendering. Rollouts are replayed in a Gaussian-splat reconstruction of the real workspace, with lighting, texture and camera augmentation, then used to fine-tune LingBot-VLA.

Results

  • Retargeting: contact angular error of 7.6 degrees, against 16.4 or more for Dex-Retargeting, SPIDER and the Wuji retargeter.
  • Realization with the arm in the loop: 95.5% rollout success, against 34.3% for ManipTrans and 24.8% for Dexplore, which are trained with a floating hand.
  • VLA in simulation: 97.5% success in seen settings and 91.25% with unseen layout, geometry and appearance. Training on data from the baselines gives at most 51.25%; dropping the grasp descriptions gives 70-80%.
  • Real robot, ten trials per grasp mode: success ranges from 20% (handle threading) to 80% (two-finger hook).

Limitations

Only grasping and pick-and-place with eight grasp modes and one hand are evaluated. The authors leave in-hand manipulation, bimanual tasks, other hand designs and tactile input to future work.

Figures

From Figure 1: From human data to robot execution. DexSeed organizes human interactions by grasp mode and expands the...
From Figure 1: From human data to robot execution. DexSeed organizes human interactions by grasp mode and expands them into simulated robot data. The VLA trained on these data executes language-specified grasps on real objects.
From Figure 2: DexSeed overview. (01) We organize human demonstrations by grasp mode and select interaction seeds. (0...
From Figure 2: DexSeed overview. (01) We organize human demonstrations by grasp mode and select interaction seeds. (02) InteractionMesh retargeting adapts these seeds to the robot hand and new object geometries. (03) Trajectory augmentation adds whole-arm motion variations, and (04) physics-based realization produces executable rollouts. (05) Gaussian Splatting rendering provides visual observations, which are paired with robot actions and grasp descriptions for VLA training.
From Figure 3: Qualitative retargeting results. (a) Comparison with baseline retargeting methods. Gap marks a separat...
From Figure 3: Qualitative retargeting results. (a) Comparison with baseline retargeting methods. Gap marks a separation between the hand and object surface. (b) Cross-object transfer to torus objects scaled by 0.75x and 1.5x, and to mugs with different geometries.
From Figure 4: Real-world VLA deployment. (a) Robot setup and test objects. (b) Grasping various objects using eight ...
From Figure 4: Real-world VLA deployment. (a) Robot setup and test objects. (b) Grasping various objects using eight grasp modes. (c) A representative manipulation sequence executed by the learned VLA.
Abstract

We study how to generate diverse, physically executable robot demonstrations for dexterous vision-language-action (VLA) learning. Existing demonstration-generation methods typically rely on costly teleoperated examples. Meanwhile, human hand--object datasets provide a reusable source, but existing transfer methods mainly reproduce given interactions. Turning these interactions into VLA training data further requires coverage of different objects and manipulation locations, together with executable arm and finger actions, and visual observations. To achieve this, we present DexSeed, a framework for low-cost generation of dexterous robot data from human hand--object interactions. Our key idea is to expand human sequences while preserving their grasps and realizing the resulting motions under arm-tracking dynamics in physics simulation. Specifically, we organize source sequences by grasp mode with shared language descriptions. InteractionMesh-based retargeting and arm trajectory augmentation adapt these sequences to the robot hand, different object geometries, and pick-and-place locations while preserving local hand--object relationships. An RL finger-control policy produces executable rollouts, while the arm tracks precomputed trajectories. Rendered observations and recorded actions from successful rollouts are paired with grasp descriptions to fine-tune a VLA. Experiments show improved contact fidelity in retargeting and more reliable whole-arm execution than the evaluated baselines. Simulation evaluation and real-world deployment further demonstrate that a VLA fine-tuned solely on the generated demonstrations can execute language-specified grasps. Project page: https://robbyant-research.github.io/DexSeed/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.