Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Taxim: An Example-based Simulation Model for GelSight Tactile Sensors

Zilin Si, Wenzhen Yuan

Seminal September 2021

TL;DR A GelSight simulator calibrated from fewer than 100 real presses. A polynomial lookup table renders the tactile images and FEM-calibrated superposition moves the markers. Its images match real ones better than three physics-based simulators (MSE 58.4 vs 90.6-215.9), and it runs at 18.1 fps on a CPU.

What they do

To simulate a GelSight sensor you must model how the soft gelpad deforms and how the internal LEDs and camera see it. Physics-based renderers for this are slow, sensor-specific and miss the real sensor’s noise. Taxim calibrates from a few real contact examples instead, and also simulates the printed markers, whose motion shows surface stretch.

How it works

  • Optical model. Example-based photometric stereo: a table maps surface normal to pixel intensity for each LED color. A second-order polynomial in image position accounts for the LEDs sitting close to the gel. It is fit from 50 presses of a 4 mm ball.
  • Deformation and shadows. Inside the contact the object’s height map is kept as-is; pyramid Gaussian kernels smooth its boundary instead of FEM. Shadows are sums of “unit” pin shadows from 10 presses of a 1 mm pin.
  • Marker motion. Node displacements depend linearly on each other through 3x3 tensors calibrated offline in ANSYS FEM. Contact nodes are corrected to virtual displacements by one matrix inversion, then superposed to move every node.

Results

  • Against real images Taxim scores L1 5.57, MSE 58.4, SSIM 0.882 and PSNR 31.0, best on all four metrics. MSE is 90.6 for the physics-based model, 123.2 for Phong’s model and 215.9 for TACTO.
  • On a CPU it runs at 18.1 fps without shadows and 9.6 fps with them. The others run at 0.1 (physics), 1.9 (TACTO) and 3.8 fps (Phong); GPU versions were not compared.
  • Marker motion error against real data is 1.02e-2 mm in magnitude and 14.57° in angle, against 1.00e-2 mm and 12.94° for FEM itself. It takes 9.22 s on a CPU, where ANSYS needs 2-4 h.
  • It runs on four GelSight sensors and a DIGIT.

Limitations

Contact is quasi-static, with no dynamics, and the marker model cannot capture partial slip under shear, a stated error source. Optical error grows for deeper and more off-center presses. Each sensor needs its own calibration (under an hour), and the FEM gelpad model does not exactly match the hand-made gel.

Figures

From Figure 1: Real and simulated GelSight images of three finely textured objects, simulated from their ground-truth...
From Figure 1: Real and simulated GelSight images of three finely textured objects, simulated from their ground-truth geometry.
From Figure 2: The pipeline: example data, calibration (polynomial table, shadow masks, gelpad FEM), then optical and...
From Figure 2: The pipeline: example data, calibration (polynomial table, shadow masks, gelpad FEM), then optical and marker motion simulation.
From Figure 8: Real tactile images next to simulations from Taxim, TACTO, Phong's model and a physics-based model.
From Figure 8: Real tactile images next to simulations from Taxim, TACTO, Phong's model and a physics-based model.
From Figure 11: A GelSight on a robot arm touches Google Scan dataset objects; most simulation artifacts come from th...
From Figure 11: A GelSight on a robot arm touches Google Scan dataset objects; most simulation artifacts come from their coarse meshes.
Abstract

Simulation is widely used in robotics for system verification and large-scale data collection. However, simulating sensors, including tactile sensors, has been a long-standing challenge. In this paper, we propose Taxim, a realistic and high-speed simulation model for a vision-based tactile sensor, GelSight. A GelSight sensor uses a piece of soft elastomer as the medium of contact and embeds optical structures to capture the deformation of the elastomer, which infers the geometry and forces applied at the contact surface. We propose an example-based method for simulating GelSight: we simulate the optical response to the deformation with a polynomial look-up table. This table maps the deformed geometries to pixel intensity sampled by the embedded camera. In order to simulate the surface markers' motion that is caused by the surface stretch of the elastomer, we apply the linear elastic deformation theory and the superposition principle. The simulation model is calibrated with less than 100 data points from a real sensor. The example-based approach enables the model to easily migrate to other GelSight sensors or its variations. To the best of our knowledge, our simulation framework is the first to incorporate marker motion field simulation that derives from elastomer deformation together with the optical simulation, creating a comprehensive and computationally efficient tactile simulation framework. Experiments reveal that our optical simulation has the lowest pixel-wise intensity errors compared to prior work and can run online with CPU computing. Our code and supplementary materials are open-sourced at https://github.com/CMURoboTouch/Taxim.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.