Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Digitizing Touch with an Artificial Multimodal Fingertip

Mike Lambeta, Tingfan Wu, Ali Sengül, Victoria Rose Most, Nolan Black, Kevin Sawyer, Romeo Mercado, Haozhi Qi, Alexander Sohn, Byron Taylor, Norb Tydingco, Gregg Kammerer, Dave Stroud, Jake Khatha, Kurt Jenkins, Kyle Most, Neal Stein, Ricardo Chavira, Thomas Craven-Bartle, Eric Sanchez, Yitian Ding, Jitendra Malik, Roberto Calandra

Seminal November 2024

TL;DR Digit 360 is a finger-shaped vision-based tactile sensor that adds audio, pressure, IMU, heat and gas sensing plus an on-device neural network accelerator. It resolves 7 µm features, predicts normal and shear force with about 1 mN median error, and runs a reflex loop with 1.2 ms event-to-action latency.

What they do

Vision-based tactile sensors like Digit and GelSight mostly capture contact geometry, at camera rates of 30-60 Hz. Digit 360, from FAIR at Meta, is a fingertip-shaped sensor that keeps a high-resolution camera view of a hemispherical gel, adds non-visual channels (vibration, pressure, heat, smell) and puts a neural network accelerator inside the finger for reflex-like control. The design is open-source.

How it works

  • Optics. A custom solid-immersion hyper-fisheye lens images the whole dome onto a 240 fps CMOS sensor with 1.1 µm pixels. The gel is a solid volume rather than a shell around an internal light pipe, which removes glints and hotspots.
  • Elastomer. A 6 µm silver film is grown chemically onto the gel, thinner than painted or airbrushed coatings. The surface scatters light at a 20-25° half-width instead of Lambertian, trading background uniformity against indentation contrast.
  • Other modalities. Microphones, MEMS pressure, IMU, temperature and gas sensors sit on a stack of modular boards.
  • Edge AI. A GAP9 accelerator (9-core RISC-V) runs models in the fingertip and can send actions straight to the robot hand.

Results

  • Force: a ResNet-50 regressor predicts normal and shear force with 1.01 mN and 1.27 mN median error at the tip (1.41 and 1.64 mN at the side), without markers. A specular finish beats a Lambertian one (1.17 vs 1.77 mN mean normal-force error).
  • Spatial resolution: features as small as 7 µm are resolved at the tip.
  • Multimodal: gas sensing identifies six household materials with 91% accuracy. With four sensors on an Allegro hand (~620k samples), IMU data dominates action classification, audio and vision dominate material classification, and all modalities together reach 85.8% average accuracy.
  • Reflex arc: 1.2 ms event-to-action on-device, vs 2.5 ms through a host and over 6 ms for Digit.

Limitations

Apart from the reflex demo, the evaluation is sensor characterization and classification of three actions and three materials; no manipulation task uses the sensor. Resolution drops away from the tip. The authors note that 240 Hz may still miss highly dynamic motion and that on-device gains may not be significant today. The paper’s latency numbers differ between sections (6 to 1.2 ms, 4 to 1 ms, 3.1 to 0.7 ms).

Figures

From Figure 1: Digit 360 on a robot hand, with the surface audio spectrograms and tactile images it records from the ...
From Figure 1: Digit 360 on a robot hand, with the surface audio spectrograms and tactile images it records from the objects it touches.
From Figure 2: Normal and shear force prediction error in three regions from the tip toward the side (median 1.01 mN ...
From Figure 2: Normal and shear force prediction error in three regions from the tip toward the side (median 1.01 mN and 1.27 mN at the tip); predicted vs. ground-truth force during one indentation; a two-pronged microindenter resolving features as small as 7 um; an exploded view of the sensor modules; images with an internal structure vs. a solid gel; and simulated surface scattering from 1 degree to Lambertian.
From Figure 3: Tapping an opaque bottle reveals its water volume through the audio peak frequency; surface audio spec...
From Figure 3: Tapping an opaque bottle reveals its water volume through the audio peak frequency; surface audio spectrograms of different textures; sensitivity to heat sources from 23 to 63 C; and gas sensing that identifies six household materials with 91% accuracy, reaching 61% within 6 seconds of approach.
From Figure 5: A reflex arc processed inside the fingertip retracts a robot hand on contact; event-to-action latency ...
From Figure 5: A reflex arc processed inside the fingertip retracts a robot hand on contact; event-to-action latency is 1.2 ms on-device vs. 2.5 ms through a host for Digit 360 and over 6 ms for Digit; force and spatial resolution compared against other tactile sensors.
Abstract

Touch is a crucial sensing modality that provides rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment (Johansson and Flanagan, 2009; Li et al., 2020; Calandra et al., 2017). However, no existing systems provide rich, multi-modal digital touch-sensing capabilities through a hemispherical compliant embodiment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities. Significantly, this fingertip contains high-resolution sensors (~8.3 million taxels) that respond to omnidirectional touch, capture multi-modal signals, and use on-device artificial intelligence to process the data in real time. Evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear forces with a resolution of 1.01 mN and 1.27 mN, respectively, perceive vibrations up to 10 kHz, sense heat, and even sense odor. Furthermore, it embeds an on-device AI neural network accelerator that acts as a peripheral nervous system on a robot and mimics the reflex arc found in humans. These results demonstrate the possibility of digitizing touch with superhuman performance. The implications are profound, and we anticipate potential applications in robotics (industrial, medical, agricultural, and consumer-level), virtual reality and telepresence, prosthetics, and e-commerce. Toward digitizing touch at scale, we open-source a modular platform to facilitate future research on the nature of touch.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.