← Paper Digest

WinDEX: Metric 3D Hand Motion Reconstruction in the Wild for Dexterous Teleoperation and Policy Learning

Hongyu Ma, Hairong Qu, Shiqi Zhao, Yongsong Yang, Kai Shi, Jiahao Ai, Xin You, Peng Yin

November 2026

Hand2Teleop, where an operator wearing CyberCap reorients a box with the robot arm and five-finger hand.

TL;DR A head-worn fisheye stereo camera with an IMU and a network trained on 1,000 hours of refined labels recover metric, world-frame 3D hand motion from everyday egocentric video. RL policies that follow these hand trajectories reach 61% simulated grasp–lift–place success, vs 8% and 17% with WiLoR and HaWoR references, and the same pipeline drives live arm–hand teleoperation.

What they do

Human hand motion can guide dexterous robots, but one camera leaves depth and scale ambiguous, and lab capture rigs don’t reach everyday scenes. WinDEX records with CyberCap, a low-cost head-worn fisheye stereo camera with an IMU. From that footage it recovers metric 3D hand poses in a gravity-aligned world frame, which then drive RL policies (Hand2Policy) and live arm–hand teleoperation (Hand2Teleop).

How it works

  • Annotation. MANO hands are initialized from WiLoR and stereo depth, then fit to human-reviewed 2D keypoints in both fisheye views. Temporal refinement switches between smooth, occlusion and rapid-motion modes. Fits with more than 15 px reprojection error are dropped, and cuVSLAM camera poses place the hands in the world frame.
  • WinDEX-Net. An end-to-end fisheye stereo network (VGGT-Omega and DINO features, ray and calibration conditioning, hand crops, odometry-conditioned temporal context) localizes and reconstructs both hands. Its predictions seed new annotations, which grew the labels from 20 h to a 1,000 h corpus from more than 200 operators.
  • Distillation. WinDEX-Window (116M parameters, 43.61 FPS on an RTX 4090) is for offline use. WinDEX-Stream (8.92M parameters, 171 FPS, causal) is for live control.
  • Robot interfaces. Wrist motion drives the arm through inverse kinematics, and finger articulation is retargeted to the hand.

Results

  • Against optical motion capture, WinDEX-Window has 15.70 mm wrist error and 16.68 mm MPJPE. The best fine-tuned baselines reach 27.45 mm (UmeTrack) and 23.48 mm (HaWoR).
  • In the world frame with cuVSLAM, joint error (W-MPJPE) is 37.29 mm vs 97.65 mm for fine-tuned HaWoR. In-the-wild reprojection error is 14.5 px vs 22.42–26.72 px for fine-tuned baselines.
  • RL on Inspire and Wuji hands averages 61% simulated grasp–lift–place success, vs 8% with WiLoR and 17% with HaWoR references. Policies transfer to a real Wuji hand on all five objects.
  • Hand2Teleop drives a 7-joint arm and a 20-joint Wuji hand at 30 Hz, tracking the operator only with the headset.

Limitations

During long occlusions the last confident pose is held, so articulation the cameras can’t see is not recovered. All training data comes from one capture device. Real-robot policy runs use privileged object and arm poses from an external camera and FoundationPose. Teleoperation is shown as task examples without success rates. WinDEX-Stream’s wrist error (42.7 mm) is well above WinDEX-Window’s 27.1 mm on the same recordings.

Videos

Recording everyday hand motion with the head-worn CyberCap stereo camera, without gloves.
Annotation, where 3D hand fits are refined against observed 2D joints and then placed in the world frame with SLAM.

Figures

From Figure 1: (a) CyberCap records stereo RGB and inertial data across many scenes for geometric–temporal annotation...
From Figure 1: (a) CyberCap records stereo RGB and inertial data across many scenes for geometric–temporal annotation and WinDEX-Net training. (b) WinDEX-D hand estimates are combined with VIO camera poses to give world-frame 3D hands. (c) Hand2Policy uses them for policy learning, and Hand2Teleop for live robot control.
From Figure 3: WinDEX-Net fuses global and local stereo features with temporal motion context to reconstruct metric h...
From Figure 3: WinDEX-Net fuses global and local stereo features with temporal motion context to reconstruct metric hand motion. Reprojecting the decoded hand sets the crops for the next refinement pass.
From Figure 12: Grasp–lift–place success for RL policies guided by WinDEX, WiLoR or HaWoR hand motion, on the Inspire...
From Figure 12: Grasp–lift–place success for RL policies guided by WinDEX, WiLoR or HaWoR hand motion, on the Inspire hand in simulation (a) and the Wuji hand in simulation and on the real robot (b). Each point is 10 trials with object positions randomized within 20 x 20 cm; shading shows 68% binomial confidence intervals.
From Figure 13: Live teleoperation with bread insertion (A), hand gestures (B), and cup grasp and box reorientation (...
From Figure 13: Live teleoperation with bread insertion (A), hand gestures (B), and cup grasp and box reorientation (C). (D) Recorded arm target and robot motion, human hand aperture, and human versus Wuji finger angles during the cup, box and gesture excerpts.
Abstract

Human manipulation data are widely used to train robots for human-like dexterity. However, low-cost, large-scale collection in the wild remains challenging. Existing methods remain limited in geometrically grounded metric-scale reconstruction, real-world generalization and inference speed. We present WinDEX, integrating (1) geometric-temporal optimization for metric 3D hand annotation from in-the-wild egocentric stereo videos, combined with visual-inertial SLAM for world-frame reconstruction; (2) WinDEX-Net, an end-to-end, fisheye-native network combining calibrated stereo, local detail and temporal context, with a data-model flywheel of model-assisted annotation and retraining; (3) distilled models for offline reconstruction and causal streaming, supporting reinforcement learning and teleoperation; and (4) low-cost CyberCap stereo-inertial capture, forming the WinDEX-Hand dataset with 1,000 hours of training annotations and held-out human-annotated in-the-wild 2D and independent optical motion-capture evaluation sets. Offline reconstruction runs at 43.61 FPS and causal streaming at 171 FPS. On held-out data, offline reconstruction achieves state-of-the-art accuracy: 14.5 pixels in-the-wild reprojection error and, with visual-inertial odometry, 37.29 mm world-frame joint error. We achieve 7.6 and 3.6 times the task success rates of reinforcement learning policies trained with WiLoR and HaWoR motion references, respectively, and demonstrate real-time arm-hand teleoperation. Project website: https://fruitrabbit.github.io/WinDEX/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.