DexHOT: Learning Hand-Object Configuration Transfer for Dexterous Manipulation

Zhewei Gui, Anthony Bolton, Jiyao Zhang, Zeyuan Chen, Qiyang Yan, Tianyu Wang, Hao Dong

November 2026

Barcode scanning, where the hand rotates a bottle to face the scanner.

TL;DR One RL policy for a 22-DoF hand learns to move between grasps that fix both the finger joints and the object's pose in the hand, then chains these short moves into longer tasks. It succeeds on 61.5% of 200 real-robot transfers and 27 of 30 demonstration-guided task trials.

What they do

Most learned dexterous policies are given an object pose as the goal, so how the hand ends up holding the object is left to chance. DexHOT makes the goal a hand-object configuration: the hand’s joint angles plus the object’s pose relative to the wrist. One policy learns to move between such configurations with the palm facing down, so the hand must keep the object from falling while it regrasps.

How it works

Configuration pools. BODex grasp synthesis on 40 objects from 40 categories, filtered in Isaac Gym, gives 825,473 stable configurations. Two configurations are compared by the mean distance between 28 hand keypoints expressed in the object frame.
Local-goal training. A recurrent policy for a 22-DoF Sharpa hand is trained only on nearby targets (20-40 mm apart), with population-based RL (EPO, 64 latent-coded blocks) across 524,288 environments with domain randomization.
Chaining. At test time a greedy planner, or a single retargeted human demonstration, supplies subgoals at 1 Hz while the frozen policy runs at 30 Hz. On hardware, FoundationPose tracks the object and a separate controller moves the Franka arm.

Results

Simulation (DexHOTBench, 120 objects): 80.5% success on seen objects, 74.4% on unseen instances and 64.4% on unseen categories, averaged over close and far transfers.
Replacing EPO with a single PPO agent drops close-transfer success to 2.5%; sampling targets from the whole pool lowers the average from 0.805 to 0.682.
Real robot: 61.5% of 200 transfers succeed on ten objects (63% seen, 60% unseen), ranging from 95% on a tennis ball down to 20% on a screwdriver or hammer.
Six demonstration-guided tasks: 27/30 trials, against 2/30 for an adapted SimToolReal and 0/30 for replaying the kinematic retargeting.

Limitations

Elongated tools wedge between fingers or swing during alignment. Other failures: slips on low-friction objects that feedback cannot catch, configurations that are only marginally stable on hardware, stalled reorientation, and FoundationPose losing track. The greedy planner does not guarantee that each step is dynamically feasible. Each task was tested five times from one fixed layout.

Videos

Screw driving, where the hand turns a drill so its bit points down.

Block assembly, where the hand reorients a half-round block to fit a slot.

Figures

From Figure 2: DexHOT overview. Top: a recurrent policy learns local transfers from physically validated configuratio... — From Figure 2: DexHOT overview. Top: a recurrent policy learns local transfers from physically validated configuration pools using population-based reinforcement learning, rewarded for configuration matching, transfer completion and small actions. Bottom: the frozen policy follows subgoals from greedy planning or human demonstrations through hierarchical chaining.

From Figure 3: DexHOTBench. (A) Object z-axis directions relative to the hand across all configuration pools, darker ... — From Figure 3: DexHOTBench. (A) Object z-axis directions relative to the hand across all configuration pools, darker green meaning denser. (B-D) Seen objects, unseen instances of training categories, and objects from unseen categories, 40 objects per split.

From Figure 4: Demonstration-guided real-robot tasks. Each row shows one task, frames ordered left to right; insets e... — From Figure 4: Demonstration-guided real-robot tasks. Each row shows one task, frames ordered left to right; insets enlarge the final hand-object configuration.

Abstract

Dexterous manipulation involves not only moving objects, but also arranging the hand around them to maintain useful contacts and enable subsequent actions. Yet many learning-based systems specify goals primarily through object poses or trajectories, leaving hand-object configurations to emerge implicitly. We introduce DexHOT, which formulates dexterous manipulation as hand-object configuration transfer: moving between feasible configurations that jointly specify hand articulation and object pose relative to the wrist. DexHOT learns a reusable relation-conditioned transfer primitive from synthetic feasible configuration pairs, without human reference motions, and composes long-horizon behaviors through closed-loop tracking of configuration subgoals. On DexHOTBench with 120 objects, a single policy achieves 80.5% transfer success on seen objects and 69.4% on unseen objects; on a real dexterous hand, it achieves 61.5% success over 200 real-robot transfers and 90% task success across six demonstration-guided tasks. These results show that explicit hand-object goals provide an expressive and composable interface for dexterous manipulation. More details and videos are available at https://dex-hot.github.io/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong