Learning Object-Centric Motion Priors from Human for Robotic Dexterous Manipulation

Zhengdong Hong, Guofeng Zhang

November 2026

TL;DR A transformer trained on human hand-object datasets predicts how the hand and object should move, and the predicted object motion becomes a task-agnostic RL reward for a dexterous hand trained in simulation and transferred zero-shot. On real robots it reaches 77% average success on YCB grasping, against 55% for the best baseline.

What they do

Rather than copying human hand motions directly, the authors train a model on human hand-object interaction data to predict how both the hand and the object will move next. The predicted object motion then serves as an RL reward for a dexterous robot hand in simulation, replacing task-specific terms such as a lifting reward. Policies transfer zero-shot to two xArm-7 arms carrying any of three different hands.

How it works

Motion predictor. A 6-layer GPT-2 transformer, conditioned on a PointNet feature of the object point cloud, autoregressively predicts future MANO hand poses and 6-DoF object poses from the last 10 steps. One model covers Dex-YCB and ARCTIC.
Starting from the real scene. A MANO hand is fitted to the robot hand’s joint positions from forward kinematics, and FoundationPose tracks the object, so prediction starts from the robot’s actual state.
Trajectory-guided RL. Predicted hand motion is retargeted to the robot. PPO in SAPIEN rewards following the retargeted hand, following the predicted object motion, thumb-plus-finger contact, and task success.
Sim-to-real. System identification of joint PID gains and force limits, plus randomized observation noise, friction, object scale and weight.

Results

Real YCB grasping (20 trials per setting): 0.77 average success vs 0.55 for AdaDexGrasp and 0.48 for HOP, and 0.80 on objects outside Dex-YCB.
Real articulated objects: 0.53 average. On everyday objects such as a real laptop, ManipTrans and Obj-Dex score 0 and this method 0.35.
Real grasping around an obstacle: 0.63 vs 0.58 for AdaDexGrasp.
With XHand1, Inspire and Ability hands, real grasping success is 69-77%. Removing the object-following reward sharply lowers sample efficiency in simulation.

Limitations

The paper has no limitations section. From its setup: policies are state-based and need an object mesh plus pose tracking; each skill policy is trained separately with hand-tuned reward weights, and system identification is redone per hand. Real success on everyday articulated objects stays at 0.35, and with Dex-YCB obstacles it roughly ties a plain reaching-reward PPO baseline (0.57 vs 0.58 in simulation).

Figures

From Figure 1: Pipeline. Observations of the initial robot and object go into a motion predictor that outputs a refer... — From Figure 1: Pipeline. Observations of the initial robot and object go into a motion predictor that outputs a reference trajectory; reinforcement learning in simulation trains a policy guided by it, which is then deployed zero-shot on the real robot, with different dexterous hands.

From Figure 2: The hand-object motion predictor. Trained on ARCTIC and Dex-YCB, it combines a PointNet feature of the... — From Figure 2: The hand-object motion predictor. Trained on ARCTIC and Dex-YCB, it combines a PointNet feature of the object point cloud with past hand and object poses in an autoregressive transformer; predicted hand poses are retargeted to the robot hand.

From Figure 3: Real-world runs of YCB grasping, articulated object manipulation and grasping with an obstacle, on see... — From Figure 3: Real-world runs of YCB grasping, articulated object manipulation and grasping with an obstacle, on seen and unseen objects, plus cross-embodiment tests with another hand.

From Figure 4: Real-world success rates with three robot hands (XHand1, Inspire, Ability) on YCB grasping, articulate... — From Figure 4: Real-world success rates with three robot hands (XHand1, Inspire, Ability) on YCB grasping, articulated objects and grasping with an obstacle.

Abstract

Manipulating diverse objects with multi-fingered dexterous hands is challenging due to the high dimensionality and complex dynamics. Human-Object Interaction (HOI) datasets provide rich knowledge about task information and embodied interactions. Instead of solely imitating the human demonstrations, our method learns to holistically predict future hand-object states by leveraging these datasets. The predicted future states of the object can serve as a general-purpose reward term for reinforcement learning, reducing reliance on task-specific reward engineering and enhancing generalization across tasks. We conduct extensive experiments on three manipulation tasks in simulation and the real world. Our approach outperforms existing SOTA methods in both success rate and generalizability on novel objects. Furthermore, we validate the cross-embodiment compatibility of our methods by successfully deploying the skills on different robot hands.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong