9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation

Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, Huazhe Xu

Seminal August 2023

Pressing a wrench on the sensor, with the tactile image, deformation representation and estimated force.

TL;DR A $15 open-source tactile sensor, 32.5 x 25.5 x 25.5 mm, that reconstructs contact shape and estimates 6D force from the raw gel image, with no markers. A DenseNet trained on 100,417 image-force pairs from 175 objects reaches 0.370 N and 0.0077 Nm mean error on unseen objects.

What they do

9DTact is a compact, open-source vision-based tactile sensor built from 3D-printed, laser-cut and off-the-shelf parts for about $15. It keeps the DTact principle (a translucent gel looks darker where contact makes it thinner) for 3D shape, and adds 6D force estimation (normal, shear and torque) without markers or painted patterns on the gel.

How it works

Hardware. A 160° camera, a board of eight current-regulated LEDs, transparent, translucent and black gel layers, and an isolation ring that blocks stray light at the edges. Black gel now also covers the sides of the translucent layer to keep out ambient light. At 20 g and 90 FPS, it is about 22% of DTact’s size.
Shape. The grayscale difference from a no-contact reference image is mapped to depth with a single-image calibration, then turned into a point cloud. A calibration board pressed into the gel imprints virtual markers for undistortion.
Force. Pressed gel flows outward and thickens around the contact, so surrounding pixels get brighter, and they gather in the direction of drag or twist. The network input stacks the darker image, the brighter image and the reference image; a DenseNet-169 regresses 6D force.
Data. 175 Thingi10K objects, 3D-printed, were pressed, dragged and twisted by hand on the sensor mounted on a 6-axis F/T sensor: 100,417 image-force pairs in about 10 hours.

Results

Shape error on 20 ball presses: MAE 0.0462 mm (Std 0.0304 mm) vs 0.0476 mm for DTact, over the full surface instead of only the center.
Force on a random 10,000-pair test split: mean error 0.307 N for forces and 0.006 Nm for torques.
On 18 held-out objects (10,422 pairs): 0.370 N and 0.0077 Nm.
After pressing sharp objects to collect over 100,000 images, imaging and illumination stayed stable and the gel showed no visible damage.

Limitations

All force data come from a single sensor, and transfer of the model to another unit is not tested. The authors note that direct comparison with other sensors is hard because test objects and reported metrics differ. No robot manipulation tasks are shown.

Videos

Pressing a set screw, with the tactile image, deformation representation and estimated force.

Pressing a star, showing the darker contact area and the brighter gel bulge around it.

Figures

From Figure 1: Exploded view and schematic of 9DTact: camera, LED board, sensor base, isolation ring, acrylic window,... — From Figure 1: Exploded view and schematic of 9DTact: camera, LED board, sensor base, isolation ring, acrylic window, sensor shell and the transparent, translucent and black gel layers.

From Figure 3: The two functions. 3D shape uses a modeling-based mapping from the difference image to a depth map and... — From Figure 3: The two functions. 3D shape uses a modeling-based mapping from the difference image to a depth map and point cloud; 6D force uses a learned network. The reference image has no contact; the tactile image shows a "9DTact" badge pressed on the gel.

From Figure 5: Shape reconstruction of M4, M6 and M8 set screws, a ball array, a micro USB port and a wheel hub model... — From Figure 5: Shape reconstruction of M4, M6 and M8 set screws, a ball array, a micro USB port and a wheel hub model. Rows: photo, tactile image, reconstructed point cloud.

From Figure 10: Test-set inputs from the standard split, with complex geometries and motions, each with its average f... — From Figure 10: Test-set inputs from the standard split, with complex geometries and motions, each with its average force and torque error.

Abstract

The advancements in vision-based tactile sensors have boosted the aptitude of robots to perform contact-rich manipulation, particularly when precise positioning and contact state of the manipulated objects are crucial for successful execution. In this work, we present 9DTact, a straightforward yet versatile tactile sensor that offers 3D shape reconstruction and 6D force estimation capabilities. Conceptually, 9DTact is designed to be highly compact, robust, and adaptable to various robotic platforms. Moreover, it is low-cost and easy-to-fabricate, requiring minimal assembly skills. Functionally, 9DTact builds upon the optical principles of DTact and is optimized to achieve 3D shape reconstruction with enhanced accuracy and efficiency. Remarkably, we leverage the optical and deformable properties of the translucent gel so that 9DTact can perform 6D force estimation without the participation of auxiliary markers or patterns on the gel surface. More specifically, we collect a dataset consisting of approximately 100,000 image-force pairs from 175 complex objects and train a neural network to regress the 6D force, which can generalize to unseen objects. To promote the development and applications of vision-based tactile sensors, we open-source both the hardware and software of 9DTact, along with a comprehensive video tutorial, all of which are available at https://linchangyi1.github.io/9DTact.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong