Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

GRASP: Gradient-Guided Sampling for Contact-Rich Physics-Based Motion Retargeting

Yang Liu, Tong Wang, Yiming Bao, Heng Yang

November 2026

Real dexterous hands replaying GRASP-optimized trajectories open loop.

TL;DR GRASP turns kinematically retargeted human motions into physically executable robot motions by sampling in simulation and steering the search with gradients fitted to past rollouts, with no differentiable simulator. It succeeds on 92.8% of 1352 bimanual OakInk2 demonstrations (best baseline 81.0%), and its refined references improve PPO tracking on dexterous hands and a Unitree G1.

What they do

Kinematic retargeting maps human motion onto a robot’s geometry, but the result need not be physically executable: a hand may follow an object without ever grasping it. GRASP is a physics-based retargeting optimizer that searches for robot commands that reproduce the demonstrated robot and object motion in simulation. It uses a single state-tracking objective without contact rewards and needs only batched rollout costs, not simulator derivatives.

How it works

  • Tempered sampling. Commands are offsets at knots around the reference. Tempered sequential Monte Carlo reweights and resamples a population of candidates toward low cost, then diversifies it with MCMC moves.
  • Gradients from archived rollouts. Every evaluated candidate, rejected ones included, is archived. A weighted quadratic fit in a low-frequency basis gives a surrogate gradient for Langevin proposals. A Metropolis–Hastings test on true simulator costs keeps the target distribution correct despite fitting error.
  • GRASP-MPC. Long clips run in receding-horizon windows with warm-started priors and reused candidate plans. For hands, the initial grasp is optimized jointly with the first window. Simulation runs in MuJoCo Warp.

Results

  • OakInk2, 1352 bimanual demos on Sharpa hands: 92.8% success, against 81.0% for Do-as-I-Do, 77.1% for SPIDER and 28.6% for kinematic retargeting. The same settings give 92.4% on Wuji and 95.0% on XHand. On 5,400 HOT3D trajectories: 81.54% ADD-AUC.
  • Unitree G1: lower joint and pelvis-translation errors than GMR and SPIDER on LAFAN1, AMASS and OMOMO (joint error 0.30 vs 0.58 for SPIDER on LAFAN1).
  • Pen-spinning PPO trained on GRASP references reaches a mean reward of 45.9 vs 21.7 and a position error of 0.99 vs 1.98 cm, while shaft-angle error rises slightly (4.3 vs 3.9 deg). G1 object-interaction policies earn higher object rewards through most of training.
  • Hardware: open-loop playback on OpenArm arms with Sharpa Wave hands (lifting a container, pouring) and closed-loop G1 crawling, box carrying and box pushing.

Limitations

The authors leave open how reliable the guidance from finite, regularized archives is, and how robust the motions and policies are to errors in object geometry and dynamics. The hardware results are demonstrations without success rates, and hand playback is open loop. On OMOMO, object rotation error (0.13 rad) is higher than SPIDER’s (0.06). A median OakInk2 clip takes about 39 min on an A100 with 1,024 particles.

Videos

Real humanoids running policies trained on GRASP-refined references.
Retargeted hand-object demonstrations in simulation.

Figures

From Figure 1: GRASP refines kinematic references from human motion through simulator feedback, combining sampling wi...
From Figure 1: GRASP refines kinematic references from human motion through simulator feedback, combining sampling with recovered gradients, for open-loop deployment and RL tracking.
From Figure 3: Recovering a smooth proposal direction from a nonsmooth rollout cost along one coordinate. (a) Current...
From Figure 3: Recovering a smooth proposal direction from a nonsmooth rollout cost along one coordinate. (a) Current particles and earlier evaluations form the archive; the cost has a kink and a jump. (b) Proximity weights favor samples near the population center. (c) The weighted quadratic fit stays differentiable across these transitions and is compared with the Gaussian-smoothed cost.
From Figure 10: Closed-loop execution of learned policies on real humanoids: (a) crawling, (b) lifting, carrying and ...
From Figure 10: Closed-loop execution of learned policies on real humanoids: (a) crawling, (b) lifting, carrying and placing a box, and (c) pushing a box along the floor.
Abstract

We study physics-based retargeting of human demonstrations for robot learning: finding robot controls that reproduce reference motions under contact dynamics. We propose GRASP (Gradient Recovery from Archived Samples for Proposals), which combines sampling-based exploration with gradient-guided refinement without requiring a differentiable simulator. GRASP-TO extends tempered sequential Monte Carlo by fitting local quadratic cost models to archived rollouts and using their gradients to guide Markov chain Monte Carlo rejuvenation. Metropolis-Hastings correction preserves the target distribution for each fixed proposal kernel, and a population analysis connects the recovered gradients to Gaussian smoothing. GRASP-MPC scales this optimizer to long demonstrations through warm-started receding-horizon planning. Experiments on dexterous hands and humanoids show accurate retargeting with a shared tracking objective and no task-specific contact rewards, including 92.8% success on 1352 bimanual OakInk2 demonstrations. The refined references improve PPO tracking in pen spinning and humanoid-object interaction. We also demonstrate dexterous manipulation and whole-body humanoid motion on real hardware.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.