Generation of GelSight Tactile Images for Sim2Real Learning

Daniel Fernandes Gomes, Paolo Paoletti, Shan Luo

Seminal January 2021

Real and simulated printers tap the same object, with the real and generated tactile images below.

TL;DR They simulate a GelSight optical tactile sensor in Gazebo by turning a depth-camera image into a smoothed elastomer heightmap and shading it with Phong's model. A classifier trained only on generated images, with texture augmentation, recognises 21 real objects with 76.19% test accuracy.

What they do

GelSight sensors image a soft, opaque-painted membrane from the inside under coloured LEDs, which gives high-resolution touch, but no simulator produced such images. The authors build a simulated GelSight in Gazebo that renders tactile images from a depth map, and compare it against a real GelSight 2014 sensor. In both worlds the sensor is mounted on a 3D printer that taps 21 small 3D-printed objects.

How it works

Elastomer heightmap. A virtual depth camera sits where the real webcam is. Its depth map is clipped at the deepest point the elastomer can reach (3 cm), then smoothed by repeated 2D Gaussian filtering (21x21 kernel, sigma 7, 6 steps), each step merged back with a max to keep sharp in-contact features. A difference of Gaussians adds the raised bump around the contact.
Illumination. Surface normals come from discrete derivatives of the heightmap; Phong’s model (ambient, diffuse, specular) shades them under four directional lights whose colours were sampled from a real tactile image.
Sensor model. Glass and elastomer are invisible to the depth camera; the elastomer has no collision geometry so objects can press into it, while the glass stops them.
Data. 99 taps per object on a 3x3x11 grid (1 mm horizontal, 0.1 mm vertical steps), collected identically in both worlds.

Results

Real vs generated images: SSIM 0.859 and MAE 8.56% after per-object alignment, against SSIM 0.826 and MAE 10.78% for their earlier workshop version.
Classifying the 21 shapes with an ImageNet-pretrained ResNet-50 backbone: 94.65% real-to-real, 82.73% sim-to-sim, 43.45% sim-to-real.
Perturbing the virtual depth maps with 12 hand-made textures resembling 3D-printing marks raises sim-to-real test accuracy to 76.19%.

Limitations

The model assumes a flat elastomer, directional lights and no shadows, so rounded or larger-area sensors need their own adaptations. Markers are not simulated, and membrane deformation is approximated by filtering a depth map rather than modelled physically. Light and smoothing parameters were set by hand and visual inspection. Reflections on squashed areas, such as the torus and round surface, still differ from the real sensor.

Videos

Figures

From Figure 1: (A) the real and (B) the simulated setup, each a GelSight sensor on a 3D printer pressing a cube with ... — From Figure 1: (A) the real and (B) the simulated setup, each a GelSight sensor on a 3D printer pressing a cube with a hollow cylinder in the centre; (C) the real tactile image and (D) the generated one.

From Figure 4: the real sensor's webcam captures RGB tactile images directly (top); the simulated sensor captures a d... — From Figure 4: the real sensor's webcam captures RGB tactile images directly (top); the simulated sensor captures a depth map and turns it into a tactile image with the proposed method (bottom).

From Figure 3: the elastomer heightmap is approximated by limiting the depth map and smoothing it with Gaussian filte... — From Figure 3: the elastomer heightmap is approximated by limiting the depth map and smoothing it with Gaussian filtering, then shaded by computing surface normals as discrete derivatives and applying Phong's illumination model.

From Figure 8: real GelSight 2014 samples (top row) and simulations with the earlier workshop method, single Gaussian... — From Figure 8: real GelSight 2014 samples (top row) and simulations with the earlier workshop method, single Gaussian and difference of Gaussians smoothing, plus a simulated GelSight 2017 sensor (last row).

Abstract

Most current works in Sim2Real learning for robotic manipulation tasks leverage camera vision that may be significantly occluded by robot hands during the manipulation. Tactile sensing offers complementary information to vision and can compensate for the information loss caused by the occlusion. However, the use of tactile sensing is restricted in the Sim2Real research due to no simulated tactile sensors being available. To mitigate the gap, we introduce a novel approach for simulating a GelSight tactile sensor in the commonly used Gazebo simulator. Similar to the real GelSight sensor, the simulated sensor can produce high-resolution images by an optical sensor from the interaction between the touched object and an opaque soft membrane. It can indirectly sense forces, geometry, texture and other properties of the object and enables Sim2Real learning with tactile sensing. Preliminary experimental results have shown that the simulated sensor could generate realistic outputs similar to the ones captured by a real GelSight sensor. All the materials used in this paper are available at https://danfergo.github.io/gelsight-simulation.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong