Sparsh: Self-supervised touch representations for vision-based tactile sensing

Carolina Higuera, Akash Sharma, Chaithanya Krishna Bodduluri, Taosha Fan, Patrick Lancaster, Mrinal Kalakrishnan, Michael Kaess, Byron Boots, Mike Lambeta, Tingfan Wu, Mustafa Mukadam

Seminal October 2024

Shear and normal force fields predicted by Sparsh (DINO) as a can slides over a DIGIT sensor.

TL;DR Self-supervised pre-training of ViT touch encoders on 460k+ unlabeled DIGIT and GelSight images, tested with frozen features on a new six-task benchmark, TacBench. Sparsh beats task- and sensor-specific models trained end-to-end by 95.1% on average when only 33-50% of the labels are available.

What they do

Vision-based tactile sensors such as DIGIT and GelSight usually get a custom model trained on labeled data for each task and sensor, and labels like force and slip need expensive lab instrumentation. Sparsh instead pre-trains general touch encoders with self-supervised learning on 460k+ unlabeled tactile images from DIGIT, GelSight 2017 and GelSight Mini. A companion benchmark, TacBench, measures how useful the frozen features are on six tasks.

How it works

Data. About 661k images from YCB-Slide, Touch-and-Go, ObjectFolder and a new Touch-Slide dataset; 70% (462.7k) are used for pre-training.
Pre-training. ViT-B encoders trained with MAE, DINO, DINOv2, I-JEPA and V-JEPA objectives. The input is two frames 5 samples apart stacked along channels (about 80 ms at 60 FPS), or a 4-frame clip for V-JEPA, with background subtraction to suppress lighting and gel differences between sensors.
Evaluation. The encoder is frozen and a small attentive probe (cross-attention plus a 2-layer MLP) is trained per task on 1-100% of the labels. The E2E baseline trains the same architecture from scratch. Tasks: force estimation, slip detection, SE(2) pose change, grasp stability, textile recognition and a bead-maze diffusion policy.

Results

Averaged over TacBench, Sparsh beats E2E by 95.1% with 33-50% of the labels. The DINO and I-JEPA variants are the strongest overall.
With a third of the labels: DIGIT force RMSE 44.0 mN (DINO) vs 61.4 mN for E2E; slip F1 0.828 (V-JEPA) vs 0.238; pose-change accuracy 0.834 (DINO) vs 0.245.
Bead maze on the real robot: Sparsh policies travel 9.4-10.8 cm before failure vs 6.7 cm for E2E.

Limitations

The pre-training data mostly contains discrete contacts with little shear-rich interaction, and the length of the tactile history is not ablated. No bead-maze policy completes the full maze on the real robot; compounding errors make the bead fall out of the fingers.

Videos

Sparsh (DINO) classifying an object's pose change as it is slid under a robot hand's DIGIT fingertips.

Figures

From Figure 1: Sparsh, a family of general touch representations, and TacBench, a benchmark of six touch-centric task... — From Figure 1: Sparsh, a family of general touch representations, and TacBench, a benchmark of six touch-centric tasks (T1-T6). Pre-trained with self-supervision on 460k+ tactile images, Sparsh works across tasks (right) and sensors (left) and outperforms task- and sensor-specific models (E2E). The middle plot uses task decoders trained on 33% of the labeled data (50% for T6).

From Figure 2: (a) New and existing tactile datasets used for pre-training, and the vision SSL methods adapted to tou... — From Figure 2: (a) New and existing tactile datasets used for pre-training, and the vision SSL methods adapted to touch: (b) Masked Autoencoder (MAE), (c) DINO/DINOv2 and (d) Image/Video Joint-Embedding Predictive Architecture (I-JEPA, V-JEPA).

From Figure 4: Sparsh versus E2E on TacBench tasks T1-T6 across amounts of labeled data. The pre-trained representati... — From Figure 4: Sparsh versus E2E on TacBench tasks T1-T6 across amounts of labeled data. The pre-trained representations outperform task- and sensor-specific models, especially when labeled data is limited.

Abstract

In this work, we introduce general purpose touch representations for the increasingly accessible class of vision-based tactile sensors. Such sensors have led to many recent advances in robot manipulation as they markedly complement vision, yet solutions today often rely on task and sensor specific handcrafted perception models. Collecting real data at scale with task centric ground truth labels, like contact forces and slip, is a challenge further compounded by sensors of various form factor differing in aspects like lighting and gel markings. To tackle this we turn to self-supervised learning (SSL) that has demonstrated remarkable performance in computer vision. We present Sparsh, a family of SSL models that can support various vision-based tactile sensors, alleviating the need for custom labels through pre-training on 460k+ tactile images with masking and self-distillation in pixel and latent spaces. We also build TacBench, to facilitate standardized benchmarking across sensors and models, comprising of six tasks ranging from comprehending tactile properties to enabling physical perception and manipulation planning. In evaluations, we find that SSL pre-training for touch representation outperforms task and sensor-specific end-to-end training by 95.1% on average over TacBench, and Sparsh (DINO) and Sparsh (IJEPA) are the most competitive, indicating the merits of learning in latent space for tactile images. Project page: https://sparsh-ssl.github.io/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong