Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation

Zilin Si, Gu Zhang, Qingwei Ben, Branden Romero, Zhou Xian, Chao Liu, Chuang Gan

Seminal March 2024

Real GelSight press-and-slide sequence, one of the system-identification recordings.

TL;DR A differentiable tactile simulator that couples an FEM model of a GelSight elastomer to MPM and PBD objects (rigid, elastic, elastoplastic, articulated, cable) through a penalty contact model. Gradient-based trajectory optimization with its tactile losses turns a block 60.8 degrees against a wall, where CMA-ES, PPO and SAC reach at most 5 degrees.

What they do

Earlier tactile simulators sit on rigid-body engines or handle only rigid objects. DiffTactile is a differentiable Taichi simulator of a GelSight sensor touching rigid, elastic, elastoplastic, articulated and cable objects. Its gradients fit simulator parameters to real sensor data and optimize manipulation trajectories.

How it works

  • Sensor. The elastomer is a Neo-Hookean FEM mesh; markers are interpolated from its deformed surface and projected through the camera model.
  • Objects and contact. MLS-MPM handles rigid, elastic and elastoplastic objects (articulated ones as rigid parts joined by soft, thin bodies), PBD handles cables. A penalty contact model with stiffness, damping and friction couples them to the sensor.
  • Optics. An MLP with positional encoding maps surface normal and viewing direction to color change, trained on 200 real 4 mm sphere indentations.
  • System ID. Adam fits the sensor’s Lamé parameters and the contact parameters to real marker and force recordings from a GelSight on a UR5e.

Results

  • System ID (real2sim marker MSE, press-slide): 3.08 px, against 3.29 for an RNN and 3.42 for CMA-ES.
  • Optics: L1 7.94 against 16.1 for Taxim’s lookup table.
  • Manipulation, 100 iterations each: with tactile losses, gradient-based optimization turns the block 60.8° (baselines at most 5.0°) and stays in contact for 51.7 along the surface (best baseline 14.0). PPO, SAC and CMA-ES leave the case lid at negative angles; gradients open it 17.1°.
  • Real world: optimized trajectories transfer to hardware, and a slip predictor trained on 100 simulated grasps (90.0% validation accuracy) lifts a deformable object that forceful and light grasps damage or let slip.

Limitations

The paper states none. In its results, tactile losses did not reduce deformation in elastoplastic grasping (0.89 vs 0.85) and hurt case opening (9.8° vs 17.1°). Forward simulation runs at 7-25 FPS on an RTX 3060, with a slower backward pass. RL baselines got only 100 iterations, which the authors say may be too few.

Videos

Simulated cable straightening: two tactile fingers grip the cable and slide along it.
Simulated object reposing: one sensor pushes a block up against the wall.
Full video (YouTube)

Figures

From Figure 2: the five simulated tasks (grasping four objects of different shapes and materials, surface following, ...
From Figure 2: the five simulated tasks (grasping four objects of different shapes and materials, surface following, cable straightening, object reposing, case opening), each shown at the start, at the end and in the tactile view.
From Figure 5: one simulation step. The FEM sensor and MPM object each run pre-contact updates, a two-way coupling de...
From Figure 5: one simulation step. The FEM sensor and MPM object each run pre-contact updates, a two-way coupling detects collisions and computes contact forces, and both use those forces in their post-contact updates.
From Figure 4: simulated tactile images next to real ones for probes of different contact geometries.
From Figure 4: simulated tactile images next to real ones for probes of different contact geometries.
From Figure 9: real-world grasp of a deformable object. A forceful grasp damages it and a light grasp slips; the sim-...
From Figure 9: real-world grasp of a deformable object. A forceful grasp damages it and a light grasp slips; the sim-to-real adaptive grasp policy lifts it.
Abstract

We introduce DIFFTACTILE, a physics-based differentiable tactile simulation system designed to enhance robotic manipulation with dense and physically accurate tactile feedback. In contrast to prior tactile simulators which primarily focus on manipulating rigid bodies and often rely on simplified approximations to model stress and deformations of materials in contact, DIFFTACTILE emphasizes physics-based contact modeling with high fidelity, supporting simulations of diverse contact modes and interactions with objects possessing a wide range of material properties. Our system incorporates several key components, including a Finite Element Method (FEM)-based soft body model for simulating the sensing elastomer, a multi-material simulator for modeling diverse object types (such as elastic, elastoplastic, cables) under manipulation, a penalty-based contact model for handling contact dynamics. The differentiable nature of our system facilitates gradient-based optimization for both 1) refining physical properties in simulation using real-world data, hence narrowing the sim-to-real gap and 2) efficient learning of tactile-assisted grasping and contact-rich manipulation skills. Additionally, we introduce a method to infer the optical response of our tactile sensor to contact using an efficient pixel-based neural module. We anticipate that DIFFTACTILE will serve as a useful platform for studying contact-rich manipulations, leveraging the benefits of dense tactile feedback and differentiable physics. Code and supplementary materials are available at the project website https://difftactile.github.io/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.