← Paper Digest

TouchTwin: Scaling Human Touch Data via Real-to-Sim-to-Real

Huiwen Han, Nuo Chen, Ziyao Zeng, Dayou Li, Hao Wang, Amir Hossein Rahmati, Xiaoning Qian, Bangya Liu, Zhiwen Fan

November 2026

A real head-camera recording beside its touch-grounded MuJoCo replay, with glove and simulated finger forces.

TL;DR Rebuild glove-recorded human manipulation in MuJoCo and let the measured touch set each finger's contact and closing force. A hand policy trained on these replays grips at 0.99x the recorded force, against 0.31x for pose imitation and 1.55x for vision-based replays.

What they do

Real-to-sim-to-real pipelines multiply human demonstrations by rebuilding them in simulation, but they rebuild the interaction from vision. A hand pose shows where each finger stops on the surface, not how hard it closes, so which fingers touch, when and how hard is lost. TouchTwin rebuilds recordings in MuJoCo with the measured human touch setting each finger’s contact and grip force, then uses each replay to generate new labeled demonstrations.

How it works

  • Input. Recordings from the TouchScale dataset, captured with head and wrist RGB-D cameras and an 880-taxel tactile glove.
  • Rebuilding the scene. Coding agents write each object as a physics-tested program, and a per-subject hand model is tracked from both cameras.
  • Touch grounding. The recorded touch closes each finger past its visible pose, setting where, when and how hard it presses. The object is never scripted and moves only through contact.
  • Data engine. Each replay is perturbed (grasp, motion speed, grip strength, object mass or friction) and re-simulated, producing successes and failures with simulated per-taxel touch. Wan2.2, conditioned on the simulation, renders replays as realistic video in new views and scenes.

Results

  • Grip force on held-out recordings: a hand policy that imitates the observed finger pose grips at 0.31x the recorded force, one trained on vision-based replays at 1.55x, and one trained on touch-grounded replays at 0.99x, succeeding in 20/20 held-out trials and 35/36 unseen perturbations.
  • A contact predictor trained on vision-based replays misses 32% of real contact frames; trained on touch-grounded replays, it misses 13%.
  • In touch-grounded replays of the mug task, the human grip is 1.6-3.7x the smallest force needed to hold the mug, which physics alone does not recover.

Limitations

All policies act on simulated assets and are tested open-loop in simulation; there are no robot-hand or real-robot experiments. The policy and contact experiments use only the mug task with small held-out sets, and the held-out clips start with the hand already in contact, so contact onset is untested. Two-hand replays still slip 12-23 mm in the hand, and generated videos inherit simulation errors.

Videos

Video generated from a simulated replay, next to the real recording and a first-frame-plus-text baseline.
Perturbed replay: mass is added to the mug during the carry until it slips out of the recorded grip.

Figures

From Figure 1: TouchTwin overview. An agentic pipeline rebuilds each recording, and the measured touch sets where, wh...
From Figure 1: TouchTwin overview. An agentic pipeline rebuilds each recording, and the measured touch sets where, when and how hard the fingers touch and checks the replay. Each replay yields perturbed demos, decoded to video in new views and scenes, that train a hand policy.
From Figure 4: Touch-grounded replays simulate the contact force. Top: the real head-camera recording and the simulat...
From Figure 4: Touch-grounded replays simulate the contact force. Top: the real head-camera recording and the simulation rendered through the same camera, for a one-hand mug move, a two-hand first-aid box carry and a graduated cylinder lifted out of a tray. Bottom: thumb, index and middle finger force measured by the glove and simulated as contact force, on the same axes. The object moves only through contact.
From Figure 6: Perturbed replays of one recording. (a) Other hand postures with the same grip effort, each with its s...
From Figure 6: Perturbed replays of one recording. (a) Other hand postures with the same grip effort, each with its simulated touch on the glove layout: four carry the mug, while a rim-wall pinch lets it swing 73 degrees in the hand. (b) Perturbations of motion, grip and object: a 12x faster carry slips the mug 19 mm, a fading grip drops it at 22% of the recorded grip, 300 g of water raises the force needed to hold it from 20.8 N to 27.4 N, 0.12x friction drops it, and a 4 Hz shake makes the hand let go.
Abstract

Human demonstrations are becoming the main source of data for dexterous manipulation, and real-to-sim-to-real pipelines multiply each demonstration in simulation. These pipelines rebuild the interaction from vision, which recovers how the hand moves. Which fingers touch, when, and how hard remain unobserved, because a hand pose shows where each finger stops on the surface rather than how far it is driven to close. This gap is a force gap: on held-out recordings, a hand policy that imitates the observed pose grips with 0.31x the recorded force, and a policy trained on physics replays of vision-tracked poses grips with 1.55x. TouchTwin closes the gap with measured human touch. Each selected TouchScale recording, captured with head and wrist RGB-D cameras and an 880-taxel glove, is rebuilt in MuJoCo. Coding agents write the objects as physics-tested programs, and a per-subject tactile hand replays the interaction with the recorded touch setting the contact and the closing force of each finger. In simulation, a policy trained on touch-grounded replays grips at 0.99x the recorded force and succeeds in 20/20 trials on held-out recordings and 35/36 unseen perturbations; a contact predictor trained on them misses 13% of real contact frames, against 32% for vision-based replays. Each replay also seeds perturbed demonstrations with simulated contact and force labels, rendered as realistic video in new views and scenes.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.