← Paper Digest

From Human Videos to Bimanual Dexterous Manipulation Data: Confidence-Guided RL Refinement of Video Priors

Yanghong Liu, Yang Chen, Kailang Xu, Bang Du, Feiyang Yang, Jiawei Xia, Zhengtao Yao, Yubiao Zhang, Masayoshi Tomizuka, Ruihai Wu, Wei Zhan

November 2026

TL;DR PriDex turns three human videos per task into bimanual dexterous robot demonstrations by refining video-derived object and wrist motion with confidence-weighted RL in simulation. Pretraining on these demonstrations raises real-robot success from 30.0% to 63.3% with the same 20 teleoperated demos.

What they do

Teleoperating demonstrations for two arms with multi-fingered hands is slow. PriDex starts from monocular human videos (three per task, from EgoDex) and turns them into robot demonstrations through RL in Isaac Sim, for a Dexmate bimanual robot with two 22-DoF Sharpa hands. It keeps where the human made contact and how the object moved, but lets the robot find its own grasp and arm motion.

How it works

  • Video priors. ViPE, HaWoR, SAM2, SAM3D and a FoundationPose-based tracker recover object meshes, 6-DoF object trajectories and wrist trajectories. A per-frame confidence score drops under hand occlusion, pose jumps or disagreement with CoTracker point tracks.
  • Grasp initialization. Contact regions from the video rank geometry-based grasp candidates from Dexonomy; cuRobo plans the arm approach and a hold test discards grasps that slip.
  • Confidence-guided RL. PPO learns residual arm corrections around an IK reference that follows the object trajectory. As confidence falls, the tracking tolerance grows from 3 to 8 cm, the residual bound from 0.05 to 0.10 rad, and a reward for following the human wrist-motion direction gains weight.
  • Distillation. Successful rollouts (200 per video) train a Diffusion Policy on RGB from one head and two wrist cameras plus proprioception.

Results

  • Simulation RL success averages 57.2% over pour, sweep, unscrew and clean, versus 2.6% and 1.5% for adapted Human2Sim2Robot and ViViDex.
  • Visuomotor policies trained on PriDex data average 31.3% success, versus 24.2% with data from the variant without wrist guidance.
  • Real robot, 10 trials per task: pretraining on 120 PriDex demos per task and then finetuning on 20 teleoperated demos raises success from 30.0% to 63.3% (pour 2 to 5, sweep 3 to 8, clean 4 to 6 out of 10).

Limitations

Wrist guidance helps on pour and unscrew but hurts sweep and clean, where the variant without it reaches higher RL success (64.7% and 99.3% versus 44.7% and 46.0%). Distilled policies stay weak on unscrewing (5.3%). The RL teachers rely on privileged simulator state, and the real-robot tests cover three tasks with fixed object placements.

Figures

From Figure 2: Overview of PriDex. From a human video it reconstructs object motion, relative wrist motion, contact r...
From Figure 2: Overview of PriDex. From a human video it reconstructs object motion, relative wrist motion, contact regions and object-motion confidence, and uses the contact cues to pick robot grasps. PPO refines these priors in simulation, and successful rollouts train a vision-based Diffusion Policy that is deployed on the real robot.
From Figure 3: The four bimanual tasks, each shown as a human demonstration (left) and the simulated robot task (righ...
From Figure 3: The four bimanual tasks, each shown as a human demonstration (left) and the simulated robot task (right): pouring, plate cleaning, sweeping and bottle-cap unscrewing.
Abstract

Collecting demonstrations for bimanual dexterous manipulation is costly, motivating the use of human videos as a source of robot training data. However, uncertainty in video-derived motion estimates and differences between human and robot embodiments make direct motion transfer challenging. In this work, we introduce PriDex, a framework that transforms human-video Priors into executable robot Dexterous manipulation demonstrations through reinforcement learning in simulation. PriDex preserves task-relevant interaction structure while allowing embodiment-specific behavior to be regenerated for the robot. Human-object interactions provide affordance cues for selecting task-relevant, robot-specific grasps from geometry-based candidates. For motion refinement, object trajectories provide a task-level reference, while relative bimanual wrist trajectories offer complementary guidance for coordinated motion. Since object-motion estimates can become unreliable under occlusion or visual ambiguity, PriDex uses object-motion confidence to adaptively balance these two priors, emphasizing object-level guidance when confidence is high and increasing the contribution of wrist-level guidance as confidence decreases. Reinforcement learning refines these video-derived priors under the robot's kinematic and physical constraints, and the resulting policies are rolled out to generate robot demonstrations for training a vision-based diffusion policy. We evaluate PriDex on four bimanual dexterous manipulation tasks, including pouring, sweeping into a dustpan, bottle-cap unscrewing, and plate cleaning, assessing both simulation-based robot data generation and downstream visuomotor policy learning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.