Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Deep Learning for Tactile Understanding From Visual and Haptic Data

Yang Gao, Lisa Anne Hendricks, Katherine J. Kuchenbecker, Trevor Darrell

Seminal November 2015

TL;DR CNNs label household objects with 24 haptic adjectives such as compressible or smooth, from BioTac touch signals, from images, or from both. On PHAC-2 the haptic CNN reaches 83.2 AUC against 78.2 for hand-designed features, and combining touch and vision reaches 85.9.

What they do

They classify objects with binary haptic adjectives (24 of them, e.g. squishy, metallic, thin) using deep networks instead of hand-designed features. Three models are trained on the PHAC-2 dataset: one on touch signals, one on images alone, so a robot can predict haptic properties without touching, and a multimodal one that uses both.

How it works

  • Data. PHAC-2 has 53 household objects, each explored ten times by two BioTac sensors on a PR2 gripper with four procedures (squeeze, hold, slow slide, fast slide), plus images from eight viewpoints.
  • Haptic CNN. Signals are normalized, downsampled to 150 steps, and the 19 electrodes compressed to 4 PCA components, giving 32 channels. Three grouped temporal convolutions keep each signal separate until a fully connected layer.
  • Visual CNN. Weights come from a GoogleNet fine-tuned for material recognition (MINC), followed by average pooling and L2 normalization, applied to an automatic crop around the object.
  • Fusion. Activations of the two trained networks are concatenated and only the final classifier is trained, with a hinge loss.

Results

  • Mean AUC over adjectives: hand-designed static+dynamic features 78.2, haptic LSTM 72.1, haptic CNN 83.2.
  • Images alone reach 71.5 from one view and 77.2 when features from eight views are combined.
  • Multimodal models score best: 84.7, and 85.9 when features from all ten haptic trials are combined.
  • The modalities are complementary: vision is better for thin, thick and cool, touch for squishy, absorbent and metallic.

Limitations

Only 53 objects (530 haptic trials), so the haptic signals are aggressively downsampled, channels are kept apart until the last layer, and the multimodal network is not fine-tuned end to end. For some objects, such as the furry eraser, fusion does worse than touch alone. Whether a simpler sensor than the BioTac would suffice is left untested.

Figures

From Figure 1: a visual CNN and a haptic CNN whose learned features are combined in a fusion layer to predict haptic ...
From Figure 1: a visual CNN and a haptic CNN whose learned features are combined in a fusion layer to predict haptic adjectives; the haptic data comes from a BioTac sensor.
From Figure 2: the haptic CNN, three temporal convolution layers and a fully connected layer feeding a haptic-adjecti...
From Figure 2: the haptic CNN, three temporal convolution layers and a fully connected layer feeding a haptic-adjective loss.
From Figure 5: the toothpaste box seen from the eight viewpoints in the dataset; it looks shiny but is not metallic.
From Figure 5: the toothpaste box seen from the eight viewpoints in the dataset; it looks shiny but is not metallic.
From Figure 6: per-adjective AUC of the best visual and best haptic classifiers, averaged over three train/test splits.
From Figure 6: per-adjective AUC of the best visual and best haptic classifiers, averaged over three train/test splits.
Abstract

Robots which interact with the physical world will benefit from a fine-grained tactile understanding of objects and surfaces. Additionally, for certain tasks, robots may need to know the haptic properties of an object before touching it. To enable better tactile understanding for robots, we propose a method of classifying surfaces with haptic adjectives (e.g., compressible or smooth) from both visual and physical interaction data. Humans typically combine visual predictions and feedback from physical interactions to accurately predict haptic properties and interact with the world. Inspired by this cognitive pattern, we propose and explore a purely visual haptic prediction model. Purely visual models enable a robot to "feel" without physical interaction. Furthermore, we demonstrate that using both visual and physical interaction signals together yields more accurate haptic classification. Our models take advantage of recent advances in deep neural networks by employing a unified approach to learning features for physical interaction and visual observations. Even though we employ little domain specific knowledge, our model still achieves better results than methods based on hand-designed features.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.