Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

MidasTouch: Monte-Carlo inference over distributions across sliding touch

Sudharshan Suresh, Zilin Si, Stuart Anderson, Michael Kaess, Mustafa Mukadam

Seminal October 2022

Real handheld sliding on a bottle while the pose distribution narrows to one mode.

TL;DR A particle filter localizes a DIGIT touch sensor sliding over a known object by matching learned codes of local 3D tactile geometry against a precomputed per-object codebook. Over 500 real trials on 10 YCB objects, the final pose error averages 1.97 cm and 21.48°.

What they do

A vision-based touch sensor sliding over an object sees only a small patch, so a single touch is ambiguous: a sharp edge could lie anywhere along a mug’s lip. MidasTouch keeps a full distribution over the sensor’s 6D pose on a known object and narrows it as the sensor slides over distinctive geometry, without visual priors. The authors also release YCB-Slide, real and simulated DIGIT sliding data on 10 YCB objects with ground-truth poses.

How it works

  • Tactile depth network. A ResNet-50 maps each DIGIT image to a heightmap (0.135 mm RMSE on simulated test data), trained on TACTO renders of 40 YCB objects with the test objects held out. The masked heightmap becomes a point cloud.
  • Tactile code network. MinkLoc3D, a sparse 3D convolution network pretrained for LIDAR place recognition and fine-tuned with a triplet loss, compresses the point cloud into a 256-d code.
  • Codebook. Per object, 50k simulated sensor poses are rendered and encoded once. A pose-space KD-tree gives any particle its code without rendering.
  • Particle filter. Robot odometry moves the particles, cosine similarity between codes sets their weights, and DBSCAN clusters them into pose hypotheses.

Results

  • Simulation, 500 trials: average final error 0.74 cm and 9.43°, from an initial spread covering the whole object. The full stack runs at about 10 Hz.
  • Real world, 500 trials with a tighter initial prior: 1.97 cm and 21.48°. Error drops on every object except the baseball, where all trials fail. Tools and intricate objects are easiest; symmetric ones are hardest.
  • Error of the closest hypothesis: median 0.28 cm and 2.03° in simulation, 1.11 cm and 10.76° in the real world.
  • 3D codes beat image embeddings on single-touch localization (normalized error 0.473 vs 0.546) with a 300x smaller codebook.

Limitations

The object must stay fixed and its mesh must be known. One contact only; breaking contact hurts the on-surface assumption, and shear is ignored. Deformable and articulated objects are out of scope.

Videos

Real sliding on the sugar box, with tactile geometry and pose error over time.
Handheld DIGIT sliding sessions on vise-clamped YCB objects for the YCB-Slide dataset.
Full video (YouTube)

Figures

From Figure 1: MidasTouch localizes a vision-based touch sensor on an object surface during sliding. Posed tactile im...
From Figure 1: MidasTouch localizes a vision-based touch sensor on an object surface during sliding. Posed tactile images go through a tactile depth network and a tactile code network, and a particle filter turns the codes into an evolving distribution of sensor pose on the object.
From Figure 2: Real DIGIT images from the YCB-Slide dataset and the 3D geometry predicted from them by the tactile de...
From Figure 2: Real DIGIT images from the YCB-Slide dataset and the 3D geometry predicted from them by the tactile depth network, which is trained in simulation.
From Figure 9: Initial and final pose error per object over 500 real-world trials from the 50 YCB-Slide trajectories.
From Figure 9: Initial and final pose error per object over 500 real-world trials from the 50 YCB-Slide trajectories.
Abstract

We present MidasTouch, a tactile perception system for online global localization of a vision-based touch sensor sliding on an object surface. This framework takes in posed tactile images over time, and outputs an evolving distribution of sensor pose on the object's surface, without the need for visual priors. Our key insight is to estimate local surface geometry with tactile sensing, learn a compact representation for it, and disambiguate these signals over a long time horizon. The backbone of MidasTouch is a Monte-Carlo particle filter, with a measurement model based on a tactile code network learned from tactile simulation. This network, inspired by LIDAR place recognition, compactly summarizes local surface geometries. These generated codes are efficiently compared against a precomputed tactile codebook per-object, to update the pose distribution. We further release the YCB-Slide dataset of real-world and simulated forceful sliding interactions between a vision-based tactile sensor and standard YCB objects. While single-touch localization can be inherently ambiguous, we can quickly localize our sensor by traversing salient surface geometries. Project page: https://suddhu.github.io/midastouch-tactile/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.