TouchTwin: Scaling Human Touch Data via Real-to-Sim-to-Real
TL;DR Rebuild glove-recorded human manipulation in MuJoCo and let the measured touch set each finger's contact and closing force. A hand policy trained on these replays grips at 0.99x the recorded force, against 0.31x for pose imitation and 1.55x for vision-based replays.
What they do
Real-to-sim-to-real pipelines multiply human demonstrations by rebuilding them in simulation, but they rebuild the interaction from vision. A hand pose shows where each finger stops on the surface, not how hard it closes, so which fingers touch, when and how hard is lost. TouchTwin rebuilds recordings in MuJoCo with the measured human touch setting each finger’s contact and grip force, then uses each replay to generate new labeled demonstrations.
How it works
- Input. Recordings from the TouchScale dataset, captured with head and wrist RGB-D cameras and an 880-taxel tactile glove.
- Rebuilding the scene. Coding agents write each object as a physics-tested program, and a per-subject hand model is tracked from both cameras.
- Touch grounding. The recorded touch closes each finger past its visible pose, setting where, when and how hard it presses. The object is never scripted and moves only through contact.
- Data engine. Each replay is perturbed (grasp, motion speed, grip strength, object mass or friction) and re-simulated, producing successes and failures with simulated per-taxel touch. Wan2.2, conditioned on the simulation, renders replays as realistic video in new views and scenes.
Results
- Grip force on held-out recordings: a hand policy that imitates the observed finger pose grips at 0.31x the recorded force, one trained on vision-based replays at 1.55x, and one trained on touch-grounded replays at 0.99x, succeeding in 20/20 held-out trials and 35/36 unseen perturbations.
- A contact predictor trained on vision-based replays misses 32% of real contact frames; trained on touch-grounded replays, it misses 13%.
- In touch-grounded replays of the mug task, the human grip is 1.6-3.7x the smallest force needed to hold the mug, which physics alone does not recover.
Limitations
All policies act on simulated assets and are tested open-loop in simulation; there are no robot-hand or real-robot experiments. The policy and contact experiments use only the mug task with small held-out sets, and the held-out clips start with the hand already in contact, so contact onset is untested. Two-hand replays still slip 12-23 mm in the hand, and generated videos inherit simulation errors.
Videos
Figures



Abstract
Human demonstrations are becoming the main source of data for dexterous manipulation, and real-to-sim-to-real pipelines multiply each demonstration in simulation. These pipelines rebuild the interaction from vision, which recovers how the hand moves. Which fingers touch, when, and how hard remain unobserved, because a hand pose shows where each finger stops on the surface rather than how far it is driven to close. This gap is a force gap: on held-out recordings, a hand policy that imitates the observed pose grips with 0.31x the recorded force, and a policy trained on physics replays of vision-tracked poses grips with 1.55x. TouchTwin closes the gap with measured human touch. Each selected TouchScale recording, captured with head and wrist RGB-D cameras and an 880-taxel glove, is rebuilt in MuJoCo. Coding agents write the objects as physics-tested programs, and a per-subject tactile hand replays the interaction with the recorded touch setting the contact and the closing force of each finger. In simulation, a policy trained on touch-grounded replays grips at 0.99x the recorded force and succeeds in 20/20 trials on held-out recordings and 35/36 unseen perturbations; a contact predictor trained on them misses 13% of real contact frames, against 32% for vision-based replays. Each replay also seeds perturbed demonstrations with simulated contact and force labels, rendered as realistic video in new views and scenes.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
