Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Binding Touch to Everything: Learning Unified Multimodal Tactile Representations

Fengyu Yang, Chao Feng, Ziyang Chen, Hyoungseob Park, Daniel Wang, Yiming Dou, Ziyao Zeng, Xien Chen, Rit Gangopadhyay, Andrew Owens, Alex Wong

Seminal January 2024

TL;DR UniTouch trains one touch encoder for several vision-based tactile sensors by aligning it to a frozen ImageBind image embedding, which links touch to text and audio without paired data. Zero-shot, it beats supervised methods at touch-to-vision retrieval on ObjectFolder 2.0 (41.9 vs 32.3 mAP).

What they do

Touch data is expensive to collect and tactile sensors differ in their outputs, so tactile representations are usually trained per sensor. UniTouch trains a single touch encoder on four visuo-tactile datasets from three sensors (GelSight, DIGIT and the Taxim simulator) by aligning touch to ImageBind’s frozen image embedding. Since that embedding is already aligned with text and audio, touch gets linked to them without any paired touch-text or touch-audio data.

How it works

  • Contrastive binding. A ViT touch encoder is trained with an InfoNCE loss to match each touch image to its paired visual image, encoded by the frozen image encoder.
  • Sensor tokens. Five learnable tokens per sensor, prepended to the patch tokens, absorb sensor-specific calibration and background color. An unseen sensor uses the tokens of the nearest sensor prototype (mean pixels, L1 distance).
  • Batch sampling. 75% of each batch comes from one dataset and the rest from others, giving hard negatives within a sensor while still mixing sensors.
  • Applications. The embedding plugs into off-the-shelf models: text prompts for zero-shot classification, Stable Diffusion unCLIP for touch-to-image, and a LLaMA adapter for a Touch-LLM.

Results

  • Material classification, linear probe: 61.3% on Touch and Go (best baseline 56.5%) and 61.2% on ObjectFolder Real, whose sensor is unseen (ImageNet features 54.8%). Zero-shot with “This feels like [CLS]”: 52.7%.
  • Grasp stability on Feeling of Success: 82.3% linear probe vs 80.3% best baseline; 65.5% zero-shot.
  • Zero-shot retrieval on ObjectFolder 2.0: 41.9, 37.9 and 38.0 mAP for touch to vision, audio and text, vs 32.3, 27.8 and 31.9 for supervised DAR.
  • Touch-LLM captions get a GPT-4 rating of 3.30/5 vs 2.33 for LLaVA-1.5. Ablation: naively training on all datasets drops zero-shot Touch and Go accuracy from 43.1% to 21.4%; sensor tokens plus sampling bring it to 52.7%.

Limitations

Only vision-based tactile sensors are covered; barometric or force sensors are out of scope. Zero-shot accuracy stays below linear probing, and touch-to-image FID (103.1) is worse than supervised Vision-from-touch (81.2). The representation is a black box that is hard to interpret.

Figures

From Figure 3: Method overview. A trainable touch encoder, given touch images and a sensor-specific token, is aligned...
From Figure 3: Method overview. A trainable touch encoder, given touch images and a sensor-specific token, is aligned with a frozen pretrained image encoder through a contrastive loss.
From Figure 4: Zero-shot image synthesis with touch. Left, an image of a scene generated from a tactile signal; right...
From Figure 4: Zero-shot image synthesis with touch. Left, an image of a scene generated from a tactile signal; right, an image stylized to match a touch signal, compared with the supervised Vision-from-touch diffusion model. Reference images are shown for illustration and are not seen by the model.
From Figure 5: Touch-LLM answering questions about touch images: grasp stability prediction, contact localization and...
From Figure 5: Touch-LLM answering questions about touch images: grasp stability prediction, contact localization and touch image captioning. Reference images are shown for illustration only.
Abstract

The ability to associate touch with other modalities has huge implications for humans and computational systems. However, multimodal learning with touch remains challenging due to the expensive data collection process and non-standardized sensor outputs. We introduce UniTouch, a unified tactile model for vision-based touch sensors connected to multiple modalities, including vision, language, and sound. We achieve this by aligning our UniTouch embeddings to pretrained image embeddings already associated with a variety of other modalities. We further propose learnable sensor-specific tokens, allowing the model to learn from a set of heterogeneous tactile sensors, all at the same time. UniTouch is capable of conducting various touch sensing tasks in the zero-shot setting, from robot grasping prediction to touch image question answering. To the best of our knowledge, UniTouch is the first to demonstrate such capabilities. Project page: https://cfeng16.github.io/UniTouch/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.