Connecting Touch and Vision via Cross-Modal Prediction

Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, Antonio Torralba

Seminal June 2019

TL;DR A KUKA arm with a GelSight sensor recorded 12,000 touches on 195 objects, and a conditional GAN given reference images and nearby frames learns to predict touch from vision and vision from touch. Raters judged its predicted touch locations to feel like the true ones 89% of the time on seen objects, close to a hand-supervised baseline at 90%.

What they do

A camera sees the whole scene, while a GelSight sensor feels about 1.5 cm x 1.5 cm. The paper poses two tasks across this scale gap, synthesizing GelSight images from video of a robot touching an object and predicting from a tactile image where the robot touches, and collects the VisGel dataset for them.

How it works

Data. A KUKA LBR iiwa with a GelSight sensor presses 195 everyday objects while a webcam films from behind: 12,000 touches, 3 million paired frames. Touch regions come from an ElasticFusion reconstruction so the arm can press without moving the object.
Model. A pix2pix-style conditional GAN with two ResNet-18 encoders, one for the input and one for reference images of the scene before contact and the sensor touching nothing, so it only has to learn what contact changes.
Rebalancing. About 60% of frames show the arm in the air, which caused mode collapse, so training pairs are sampled by a rarity score: the variance of the Laplacian of the difference from the reference tactile image.
Temporal cues. The input is 5 grayscale frames spaced 2 apart, keeping predicted contact in sync.

Results

Vision to touch: AMT raters judged the output real 46.6% of the time on seen objects and 38.2% on unseen ones, against 28.1% and 21.7% for pix2pix and 40.0% and 34.9% without rebalancing.
Without temporal input, models miss or shift the moment of contact and have the largest marker-deformation error.
Touch to vision: raters said the predicted touch location feels like the true one 89.2% (seen) and 83.4% (unseen) of the time, vs 44.5% and 25.2% for pix2pix and 90.4% and 85.3% for a Stacked Hourglass network trained on 1,000 hand-labeled positions.

Limitations

Both directions need reference images of the untouched scene and sensor, filmed from one fixed webcam; small objects the arm occludes were dropped. Touch-to-vision images fool raters only 30.5% (seen) and 24.2% (unseen) of the time, below mode-collapsed pix2pix on unseen objects (26.1%), and temporal cues did not always help there. Every metric is worse on unseen objects.

Figures

From Figure 1: A robot arm with a GelSight sensor touches objects while a webcam films the scene (a, b). Given refere... — From Figure 1: A robot arm with a GelSight sensor touches objects while a webcam films the scene (a, b). Given reference images, the model predicts touch from vision (c) and vision from touch (d).

From Figure 3: The vision-to-touch model. Two ResNet encoders take a sequence of video frames and the reference visio... — From Figure 3: The vision-to-touch model. Two ResNet encoders take a sequence of video frames and the reference vision and touch images; a decoder predicts the tactile image, and a discriminator tells it apart from real tactile data.

From Figure 4: Vision-to-touch (a) and touch-to-vision (b) predictions with and without reference images, next to the... — From Figure 4: Vision-to-touch (a) and touch-to-vision (b) predictions with and without reference images, next to the ground truth.

From Figure 6: Vision-to-touch errors in detecting the moment of contact (top) and in marker deformation (bottom), on... — From Figure 6: Vision-to-touch errors in detecting the moment of contact (top) and in marker deformation (bottom), on seen and unseen objects.

Abstract

Humans perceive the world using multi-modal sensory inputs such as vision, audition, and touch. In this work, we investigate the cross-modal connection between vision and touch. The main challenge in this cross-domain modeling task lies in the significant scale discrepancy between the two: while our eyes perceive an entire visual scene at once, humans can only feel a small region of an object at any given moment. To connect vision and touch, we introduce new tasks of synthesizing plausible tactile signals from visual inputs as well as imagining how we interact with objects given tactile data as input. To accomplish our goals, we first equip robots with both visual and tactile sensors and collect a large-scale dataset of corresponding vision and tactile image sequences. To close the scale gap, we present a new conditional adversarial model that incorporates the scale and location information of the touch. Human perceptual studies demonstrate that our model can produce realistic visual images from tactile data and vice versa. Finally, we present both qualitative and quantitative experimental results regarding different system designs, as well as visualizing the learned representations of our model.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong