WinDEX: Metric 3D Hand Motion Reconstruction in the Wild for Dexterous Teleoperation and Policy Learning
TL;DR A head-worn fisheye stereo camera with an IMU and a network trained on 1,000 hours of refined labels recover metric, world-frame 3D hand motion from everyday egocentric video. RL policies that follow these hand trajectories reach 61% simulated grasp–lift–place success, vs 8% and 17% with WiLoR and HaWoR references, and the same pipeline drives live arm–hand teleoperation.
What they do
Human hand motion can guide dexterous robots, but one camera leaves depth and scale ambiguous, and lab capture rigs don’t reach everyday scenes. WinDEX records with CyberCap, a low-cost head-worn fisheye stereo camera with an IMU. From that footage it recovers metric 3D hand poses in a gravity-aligned world frame, which then drive RL policies (Hand2Policy) and live arm–hand teleoperation (Hand2Teleop).
How it works
- Annotation. MANO hands are initialized from WiLoR and stereo depth, then fit to human-reviewed 2D keypoints in both fisheye views. Temporal refinement switches between smooth, occlusion and rapid-motion modes. Fits with more than 15 px reprojection error are dropped, and cuVSLAM camera poses place the hands in the world frame.
- WinDEX-Net. An end-to-end fisheye stereo network (VGGT-Omega and DINO features, ray and calibration conditioning, hand crops, odometry-conditioned temporal context) localizes and reconstructs both hands. Its predictions seed new annotations, which grew the labels from 20 h to a 1,000 h corpus from more than 200 operators.
- Distillation. WinDEX-Window (116M parameters, 43.61 FPS on an RTX 4090) is for offline use. WinDEX-Stream (8.92M parameters, 171 FPS, causal) is for live control.
- Robot interfaces. Wrist motion drives the arm through inverse kinematics, and finger articulation is retargeted to the hand.
Results
- Against optical motion capture, WinDEX-Window has 15.70 mm wrist error and 16.68 mm MPJPE. The best fine-tuned baselines reach 27.45 mm (UmeTrack) and 23.48 mm (HaWoR).
- In the world frame with cuVSLAM, joint error (W-MPJPE) is 37.29 mm vs 97.65 mm for fine-tuned HaWoR. In-the-wild reprojection error is 14.5 px vs 22.42–26.72 px for fine-tuned baselines.
- RL on Inspire and Wuji hands averages 61% simulated grasp–lift–place success, vs 8% with WiLoR and 17% with HaWoR references. Policies transfer to a real Wuji hand on all five objects.
- Hand2Teleop drives a 7-joint arm and a 20-joint Wuji hand at 30 Hz, tracking the operator only with the headset.
Limitations
During long occlusions the last confident pose is held, so articulation the cameras can’t see is not recovered. All training data comes from one capture device. Real-robot policy runs use privileged object and arm poses from an external camera and FoundationPose. Teleoperation is shown as task examples without success rates. WinDEX-Stream’s wrist error (42.7 mm) is well above WinDEX-Window’s 27.1 mm on the same recordings.
Videos
Figures




Abstract
Human manipulation data are widely used to train robots for human-like dexterity. However, low-cost, large-scale collection in the wild remains challenging. Existing methods remain limited in geometrically grounded metric-scale reconstruction, real-world generalization and inference speed. We present WinDEX, integrating (1) geometric-temporal optimization for metric 3D hand annotation from in-the-wild egocentric stereo videos, combined with visual-inertial SLAM for world-frame reconstruction; (2) WinDEX-Net, an end-to-end, fisheye-native network combining calibrated stereo, local detail and temporal context, with a data-model flywheel of model-assisted annotation and retraining; (3) distilled models for offline reconstruction and causal streaming, supporting reinforcement learning and teleoperation; and (4) low-cost CyberCap stereo-inertial capture, forming the WinDEX-Hand dataset with 1,000 hours of training annotations and held-out human-annotated in-the-wild 2D and independent optical motion-capture evaluation sets. Offline reconstruction runs at 43.61 FPS and causal streaming at 171 FPS. On held-out data, offline reconstruction achieves state-of-the-art accuracy: 14.5 pixels in-the-wild reprojection error and, with visual-inertial odometry, 37.29 mm world-frame joint error. We achieve 7.6 and 3.6 times the task success rates of reinforcement learning policies trained with WiLoR and HaWoR motion references, respectively, and demonstrate real-time arm-hand teleoperation. Project website: https://fruitrabbit.github.io/WinDEX/.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
