TactileLab: Sim-to-Real Dexterous Robot Manipulation with Efficient Shear-Sensitive Tactile Simulation

Yijiong Lin, Wenhao He, Toby Jack, Max Yang, Haoran Li, Nathan F. Lepora

November 2026

Real dual-arm object following, the tactile follower arm keeping contact with a cube moved by the leader arm.

TL;DR TactileLab adds simulated tactile flow, a dense shear field computed from rigid-body contact motion in IsaacLab, so RL policies can feel tangential contact at scale. Trained only in simulation, a LEAP Hand policy using contact depth plus flow makes 1.95 real in-hand rotations in 30 s, against 0.82 with proprioception alone.

What they do

Rigid-body tactile simulators are fast but give only contact depth, which misses shear. Soft-body simulators capture shear but are too slow for large-scale RL. TactileLab, built on IsaacLab, adds simulated tactile flow: a dense 2D field of how each contact point moves tangentially across the sensor, computed from rigid-body contact kinematics. Policies train on depth and flow in simulation and run on real robots by translating real tactile images into the same representation.

How it works

Tactile flow. Contact-patch points are projected onto the image plane of a camera modelled inside the sensor. Their image-space displacement over one timestep of sensor-object relative motion is the flow, zero outside contact. No object SDF or elastomer model is needed.
Policy learning. A teacher trained with RL on privileged state (object pose, mass, friction, perturbations) is distilled into a student. Its adaptation module, PETS-Net, combines positional-encoded proprioception-action histories with CNN-encoded depth and flow histories through a TCN.
Real-to-sim. TacFlowNet, a pix2pix-style conditional GAN, maps two consecutive real tactile images to simulated flow. It is trained on paired real and simulated pressing, sliding, tilting and twisting motions.

Results

Simulated in-hand rotation under perturbations: depth plus flow has the lowest fall rate, 11.76% in distribution and 18.69% on unseen objects, against 47.66% and 57.85% for proprioception alone.
Shear estimation from six real sensors: for a markerless GelSight Mini, translating to simulated flow cuts translational error from 1.79 to 0.34 mm/s. DIGIT stays weakest (3.33 °/s rotational error).
Real dual-arm following, trained on a rigid cube moving in straight lines: 0.91 mm / 0.019 rad error on that cube, up to 6.43 mm / 0.057 rad on a foam brain moving in circles.
Real LEAP Hand with four tactile fingertips, trained only on capsules: 1.95 rotations in 30 s with depth plus flow, against 1.70 depth only, 1.61 low-dimensional contact and 0.82 proprioception. On cube-shaped objects: 1.33 against 0.28 for low-dimensional contact.

Limitations

Real-to-sim transfer assumes sticking contact; disturbances that cause slip can lead to unrecoverable states, and slip is not simulated. Handling more diverse object shapes would need larger real-to-sim datasets and geometry-aware transfer models.

Videos

Simulated in-hand rotation and dual-arm tasks with their depth and tactile-flow streams, trained in parallel environments.

Simulated in-hand rotation with each fingertip's contact depth and tactile flow shown alongside.

Figures

From Figure 1: TactileLab overview: GPU-accelerated tactile simulation in IsaacLab with the simulated tactile flow mo... — From Figure 1: TactileLab overview: GPU-accelerated tactile simulation in IsaacLab with the simulated tactile flow modality, tactile policy and representation learning, a real-to-sim transfer pipeline for several real tactile sensors, and sim-to-real deployment on different robots.

From Figure 2: Computing simulated tactile flow. The 3D contact patch is projected onto the tactile image plane, and ... — From Figure 2: Computing simulated tactile flow. The 3D contact patch is projected onto the tactile image plane, and the image-space displacement caused by the relative motion between sensor and object gives a dense flow field; contact depth alone cannot tell sliding from twisting.

From Figure 3: (a) Multimodal tactile policy learning for in-hand manipulation, with unknown random force and torque ... — From Figure 3: (a) Multimodal tactile policy learning for in-hand manipulation, with unknown random force and torque perturbations applied to the object. (b) The PETS-Net architecture for high-dimensional tactile policy learning.

From Figure 13: Snapshots of real-world in-hand rotation of a toy cylinder.

Abstract

Robotic simulators that represent shear within tactile sensing are becoming increasingly important for contact-rich manipulation. However, simulating contact shear with high-fidelity deformable or soft-body simulation is computationally demanding, limiting scalability to large-scale reinforcement learning. Here we introduce \textit{TactileLab}, a scalable multimodal tactile simulation framework built on IsaacLab, with a particular focus on a new optical-flow-based tactile representation termed \textit{simulated tactile flow}. Instead of relying on soft-body physics to model shear deformation, TactileLab uses rigid-body simulation to efficiently estimate dense tangential motion at the contact interface, producing tactile flow fields that encode shear-relevant contact dynamics. This flow field complements geometric features from a contact depth field. We further introduce a novel policy architecture for jointly processing long-horizon proprioceptive signals and rich tactile observations. Across multiple contact-rich tasks, policies trained using simulated tactile flow learn efficiently at scale and exhibit emergent contact-aware behaviours. To facilitate sim-to-real policy deployment, we then propose a novel real-to-sim tactile translation method that maps real tactile image streams to simulated tactile flow and depth fields, together with a new tactile data collection method, and validate it across diverse tactile sensor designs. Using this pipeline, we demonstrate real-world deployment on tasks encompassing dual-arm and dexterous in-hand manipulation. Overall, TactileLab provides an end-to-end sim-to-real framework for scalable tactile policy learning, enabling diverse shear-sensitive manipulation across heterogeneous tactile robotic systems.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong