Touch and Go: Learning from Human-Collected Vision and Touch
TL;DR People walked through indoor and outdoor scenes pressing a GelSight sensor onto objects while filming, collecting 13.9k touches of about 3971 objects. Tactile features learned from these image-touch pairs beat ImageNet features at material recognition (54.7% vs 46.9%) and robot grasp-outcome prediction (78.1% vs 73.0%).
What they do
Earlier visuo-tactile datasets were collected by robots on lab tabletops or in simulation, so they cover a small number of graspable, mostly rigid objects. In Touch and Go, people collect the data instead. One person presses a GelSight sensor onto objects in classrooms, hallways, apartments, hiking trails, playgrounds and streets while a second person films the hand with a webcam. The authors then use the dataset for three tasks: learning tactile features, restyling images to match a touch signal, and predicting future touch.
How it works
- Collection. A GelSight sensor about 1.5 cm across and a webcam both record at about 27 Hz to one laptop. Collectors press each object about 3 times, for 0.7 s on average, and aim for parts with complex geometry. No separate force sensor is used.
- Annotation. A ResNet-18 press detector, trained on 10k hand-labeled frames, finds the frames in contact (97% test accuracy, about 66% chance), and crowd workers correct its errors. Five workers label each pressed frame with one of 20 material categories.
- Applications. Contrastive Multiview Coding learns matched image and touch embeddings. For stylization, a CUT-based generator restyles an image conditioned on a GelSight frame, and a discriminator judges whether the resulting image-touch pair would co-occur. For touch prediction, a 3D-convolution residual network uses separate touch and vision encoders.
Results
- The dataset holds about 13.9k touches of about 3971 objects (estimated from counts in 10% of the videos), split 52.2% indoor and 47.8% outdoor. That is about 4x the object instances of ObjectFolder 2.0.
- Linear probes on frozen features: 78.1% grasp-outcome accuracy on Feeling of Success data (ImageNet 73.0%, VisGel-trained 75.6%), 54.7% on material (ImageNet 46.9%), 77.3% on hard/soft and 79.4% on rough/smooth.
- Stylization: 0.197 visuo-tactile similarity, vs 0.178 for a CycleGAN variant and 0.165 for unedited images.
- Touch prediction: adding vision helps both models. For example, LPIPS for Geng et al. drops from 0.103 to 0.091.
Limitations
The data comes mostly from around the University of Michigan campus and from a few collectors, so it may not generalize to other places. Collectors may probe unrepresentative parts of objects and do not press with consistent force. The robot result is offline: a linear classifier predicts grasp outcomes on a previously recorded dataset.
Videos
Figures
Abstract
The ability to associate touch with sight is essential for tasks that require physically interacting with objects in the world. We propose a dataset with paired visual and tactile data called Touch and Go, in which human data collectors probe objects in natural environments using tactile sensors, while simultaneously recording egocentric video. In contrast to previous efforts, which have largely been confined to lab settings or simulated environments, our dataset spans a large number of "in the wild" objects and scenes. To demonstrate our dataset's effectiveness, we successfully apply it to a variety of tasks: 1) self-supervised visuo-tactile feature learning, 2) tactile-driven image stylization, i.e., making the visual appearance of an object more consistent with a given tactile signal, and 3) predicting future frames of a tactile signal from visuo-tactile inputs.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.