From Human Videos to Bimanual Dexterous Manipulation Data: Confidence-Guided RL Refinement of Video Priors

TL;DR PriDex turns three human videos per task into bimanual dexterous robot demonstrations by refining video-derived object and wrist motion with confidence-weighted RL in simulation. Pretraining on these demonstrations raises real-robot success from 30.0% to 63.3% with the same 20 teleoperated demos.
What they do
Teleoperating demonstrations for two arms with multi-fingered hands is slow. PriDex starts from monocular human videos (three per task, from EgoDex) and turns them into robot demonstrations through RL in Isaac Sim, for a Dexmate bimanual robot with two 22-DoF Sharpa hands. It keeps where the human made contact and how the object moved, but lets the robot find its own grasp and arm motion.
How it works
- Video priors. ViPE, HaWoR, SAM2, SAM3D and a FoundationPose-based tracker recover object meshes, 6-DoF object trajectories and wrist trajectories. A per-frame confidence score drops under hand occlusion, pose jumps or disagreement with CoTracker point tracks.
- Grasp initialization. Contact regions from the video rank geometry-based grasp candidates from Dexonomy; cuRobo plans the arm approach and a hold test discards grasps that slip.
- Confidence-guided RL. PPO learns residual arm corrections around an IK reference that follows the object trajectory. As confidence falls, the tracking tolerance grows from 3 to 8 cm, the residual bound from 0.05 to 0.10 rad, and a reward for following the human wrist-motion direction gains weight.
- Distillation. Successful rollouts (200 per video) train a Diffusion Policy on RGB from one head and two wrist cameras plus proprioception.
Results
- Simulation RL success averages 57.2% over pour, sweep, unscrew and clean, versus 2.6% and 1.5% for adapted Human2Sim2Robot and ViViDex.
- Visuomotor policies trained on PriDex data average 31.3% success, versus 24.2% with data from the variant without wrist guidance.
- Real robot, 10 trials per task: pretraining on 120 PriDex demos per task and then finetuning on 20 teleoperated demos raises success from 30.0% to 63.3% (pour 2 to 5, sweep 3 to 8, clean 4 to 6 out of 10).
Limitations
Wrist guidance helps on pour and unscrew but hurts sweep and clean, where the variant without it reaches higher RL success (64.7% and 99.3% versus 44.7% and 46.0%). Distilled policies stay weak on unscrewing (5.3%). The RL teachers rely on privileged simulator state, and the real-robot tests cover three tasks with fixed object placements.
Figures


Abstract
Collecting demonstrations for bimanual dexterous manipulation is costly, motivating the use of human videos as a source of robot training data. However, uncertainty in video-derived motion estimates and differences between human and robot embodiments make direct motion transfer challenging. In this work, we introduce PriDex, a framework that transforms human-video Priors into executable robot Dexterous manipulation demonstrations through reinforcement learning in simulation. PriDex preserves task-relevant interaction structure while allowing embodiment-specific behavior to be regenerated for the robot. Human-object interactions provide affordance cues for selecting task-relevant, robot-specific grasps from geometry-based candidates. For motion refinement, object trajectories provide a task-level reference, while relative bimanual wrist trajectories offer complementary guidance for coordinated motion. Since object-motion estimates can become unreliable under occlusion or visual ambiguity, PriDex uses object-motion confidence to adaptively balance these two priors, emphasizing object-level guidance when confidence is high and increasing the contribution of wrist-level guidance as confidence decreases. Reinforcement learning refines these video-derived priors under the robot's kinematic and physical constraints, and the resulting policies are rolled out to generate robot demonstrations for training a vision-based diffusion policy. We evaluate PriDex on four bimanual dexterous manipulation tasks, including pouring, sweeping into a dustpan, bottle-cap unscrewing, and plate cleaning, assessing both simulation-based robot data generation and downstream visuomotor policy learning.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
