Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projections

Yashraj Narang, Balakumar Sundaralingam, Miles Macklin, Arsalan Mousavian, Dieter Fox

Seminal March 2021

TL;DR A finite-element model of the BioTac fingertip in GPU-based Isaac Gym matches a validated ANSYS model at 75x the speed. Small networks that project between learned latent spaces of simulated skin deformation and real electrode signals then predict real BioTac output for unseen objects and recover contact patches from real signals.

What they do

The BioTac turns skin deformation into fluidic impedances read at 19 electrodes, which is hard to simulate. The authors rebuild their experimentally validated ANSYS finite-element (FEM) model of the BioTac in Isaac Gym. They do not simulate the electrodes; instead they learn a mapping between simulated deformations and real electrode signals.

How it works

  • FEM model. A roughly 4000-node mesh of the skin around a rigid core, linear-elastic and without the internal fluid. Stiffness, Poisson’s ratio and friction are fitted to the ANSYS force profile of one indentation.
  • Self-supervised latents. A graph-convolutional VAE compresses meshes to 128 dimensions and a fully connected VAE compresses electrode signals to 8. Both train on unlabeled contacts with 6 pegs and 3 table features.
  • Latent projection. With the VAEs frozen, two small MLPs map between the latents in both directions, trained on 359 labeled indentations.

Results

  • Over 358 held-out indentations from 8 indenters, the mean force error against ANSYS is 0.153 N.
  • 359 indentations take 33 min on 1 GPU versus about 42 h on 6 CPUs in ANSYS.
  • Median electrode RMS error is 0.010 for latent projection vs 0.012 for a fully supervised PointNet++ on unseen trajectories, and 0.015 vs 0.016 on the unseen ring indenter (31 vs 32 raw units of 4095). A prior method reported 305 units on unseen objects.
  • The inverse projection recovers contact patches from real signals, judged visually. For the ring it misses bimodal patches, which the authors attribute to the mesh VAE bottleneck.

Limitations

One 6 mm indentation still takes about 5.6 s to simulate, too slow for model-predictive control. Evaluation used a single BioTac, and manufacturing variation may require data from more sensors. In the paper, unseen-object results cover only the ring.

Figures

From Figure 2: A-D) Cross-sections of the 4 BioTac geometries tried: deformable shell, deformable shell with rigid co...
From Figure 2: A-D) Cross-sections of the 4 BioTac geometries tried: deformable shell, deformable shell with rigid core, deformable solid with rigid core, and deformable solid. E) Configuration B tuned in Isaac Gym to reproduce the ANSYS force profile of one shear-rich indentation, with a mean force error of 0.125 N.
From Figure 3: Graph convolutional autoencoders compress 4000-node FEM meshes to a 128-dim latent, and fully connecte...
From Figure 3: Graph convolutional autoencoders compress 4000-node FEM meshes to a 128-dim latent, and fully connected autoencoders compress the 19 electrode signals to an 8-dim latent. Forward and inverse projections between the two latents are trained on a small supervised dataset.
From Figure 6: A) Ground-truth (green) and predicted (red) signals of 2 electrodes over many contacts with an unseen ...
From Figure 6: A) Ground-truth (green) and predicted (red) signals of 2 electrodes over many contacts with an unseen object, for the fully supervised baseline, latent projection and the electrode VAE. B) RMS error over all electrodes for unseen trajectories and unseen objects. C) Coverage plot of L1 distance to ground truth.
From Figure 8: Top) Free-form real contacts with pegs and table features (corner, edge, surface). Bottom) Contact pat...
From Figure 8: Top) Free-form real contacts with pegs and table features (corner, edge, surface). Bottom) Contact patches predicted from the real electrode signals, colored by displacement from the undeformed skin (0-2 mm).
Abstract

Tactile sensing is critical for robotic grasping and manipulation of objects under visual occlusion. However, in contrast to simulations of robot arms and cameras, current simulations of tactile sensors have limited accuracy, speed, and utility. In this work, we develop an efficient 3D finite element method (FEM) model of the SynTouch BioTac sensor using an open-access, GPU-based robotics simulator. Our simulations closely reproduce results from an experimentally-validated model in an industry-standard, CPU-based simulator, but at 75x the speed. We then learn latent representations for simulated BioTac deformations and real-world electrical output through self-supervision, as well as projections between the latent spaces using a small supervised dataset. Using these learned latent projections, we accurately synthesize real-world BioTac electrical output and estimate contact patches, both for unseen contact interactions. This work contributes an efficient, freely-accessible FEM model of the BioTac and comprises one of the first efforts to combine self-supervision, cross-modal transfer, and sim-to-real transfer for tactile sensors.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.