TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors

Shaoxiong Wang, Mike Lambeta, Po-Wei Chou, Roberto Calandra

Seminal December 2020

TL;DR TACTO is an open-source simulator that renders camera images from vision-based tactile sensors like DIGIT and OmniTact by syncing object poses from PyBullet into an OpenGL scene, reaching 200 frames per second for one DIGIT on GPU. The authors use it to collect 1 million simulated grasps for grasp-stability learning and to learn a marble-rolling controller in 8 minutes.

What they do

Vision-based tactile sensors such as DIGIT and OmniTact are cameras looking at a gel, so simulating them needs both contact and optical rendering, fast enough to be useful for learning. TACTO is an open-source simulator for these sensors. It handles the rendering, and leaves contact dynamics to the physics engine (PyBullet by default).

How it works

Gel and object meshes are loaded into a Pyrender (OpenGL) scene once. Each step TACTO copies link poses from PyBullet and re-renders depth and RGB. Rebuilding a mesh from a depth map every frame instead capped speed at 20 fps.
A configuration file defines each sensor: gel pose and mesh, cameras, lights, noise and the force-to-deformation mapping. OmniTact, with a curved surface, 5 cameras and 11 lights, is set up this way.
A piecewise-linear function maps PyBullet normal force to gel deformation depth, with lower and upper thresholds.
Renderings can be calibrated with real sensor readings, and shadows cost about 2 ms extra on GPU.

Results

Speed: one DIGIT at 160x120 renders at 220 fps with one object and 80 fps with 100 objects (10 in contact) on an RTX 2080 Super; 60 fps on CPU. Four DIGITs on an Allegro hand run at 50 fps.
Grasp stability: 1 million grasps collected in one day on 5 threads. Touch learns from little data while vision needs 3-4 orders of magnitude more; two sensors work much better than one.
Marble rolling: Bayesian optimization learns a linear controller in 8 minutes (50 iterations).
Sim2Real pen pose estimation: training on simulation with color jitter gives 1.66 mm / 11.6° error, close to 64 real samples (2.01 mm / 10.96°). Simulation plus 128 real samples reaches 0.52 mm / 4.14°, against 0.76 mm / 4.96° for the 128 real samples alone.

Limitations

Depth and RGB come from one rigid render, so gel deformation at the contact boundary is not modeled and sharp edges look wrong. Contact forces come from PyBullet’s rigid-body model, and deformable objects need the slower render-from-depth path. Grasping uses a single box, and Sim2Real is a proof of concept on one task.

Figures

From Figure 1: TACTO renders tactile readings for sensors on a four-finger hand, on a parallel-jaw gripper and on a d... — From Figure 1: TACTO renders tactile readings for sensors on a four-finger hand, on a parallel-jaw gripper and on a dome-shaped sensor touching a ball, at over 100 Hz.

From Figure 2: TACTO sits between the physics simulator and the Pyrender rendering engine, and loads a configuration ... — From Figure 2: TACTO sits between the physics simulator and the Pyrender rendering engine, and loads a configuration file for each sensor design (DIGIT, OmniTact).

From Figure 10: A successful and a failed simulated grasp, with the two DIGIT readings at each stage. In the failure ... — From Figure 10: A successful and a failed simulated grasp, with the two DIGIT readings at each stage. In the failure the box is held only by its corner and slips out after lifting.

From Figure 12: Simulated (top) and real (bottom) tactile readings of a pen, with the estimated contact pose drawn as... — From Figure 12: Simulated (top) and real (bottom) tactile readings of a pen, with the estimated contact pose drawn as a red line.

Abstract

Simulators perform an important role in prototyping, debugging, and benchmarking new advances in robotics and learning for control. Although many physics engines exist, some aspects of the real world are harder than others to simulate. One of the aspects that have so far eluded accurate simulation is touch sensing. To address this gap, we present TACTO - a fast, flexible, and open-source simulator for vision-based tactile sensors. This simulator allows to render realistic high-resolution touch readings at hundreds of frames per second, and can be easily configured to simulate different vision-based tactile sensors, including DIGIT and OmniTact. In this paper, we detail the principles that drove the implementation of TACTO and how they are reflected in its architecture. We demonstrate TACTO on a perceptual task, by learning to predict grasp stability using touch from 1 million grasps, and on a marble manipulation control task. Moreover, we provide a proof-of-concept that TACTO can be successfully used for Sim2Real applications. We believe that TACTO is a step towards the widespread adoption of touch sensing in robotic applications, and to enable machine learning practitioners interested in multi-modal learning and control. TACTO is open-source at https://github.com/facebookresearch/tacto.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong