See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation

Hao Li, Yizhi Zhang, Junzhe Zhu, Shaoxiong Wang, Michelle A Lee, Huazhe Xu, Edward Adelson, Li Fei-Fei, Ruohan Gao, Jiajun Wu

Seminal December 2022

TL;DR A robot arm with a camera, a contact microphone and GelSight fingertip sensors fuses all three senses with self-attention and learns dense packing and pouring from demonstrations. With all three it inserts a peg past hidden bumps in 100% of trials (vision alone 25%) and pours to within 1.41 g of a 40 g target.

What they do

A Franka Panda arm sees with a third-person camera, hears with a piezo contact microphone attached to the object of interest, and feels with GelSight sensors on its fingertips. A self-attention model fuses the three streams and is trained by behavior cloning on demonstrations commanded from a keyboard. The paper tests it on dense packing and pouring and studies what each sense contributes.

How it works

Encoders. A ResNet-18 embeds each camera image, mel spectrogram (0.5 s of audio) and tactile image, with a history of N=6 observations per modality spanning 3 s.
Fusion. Multi-head self-attention runs over all these features, so each can attend across modalities, across time, or both. The outputs are concatenated and an MLP predicts a discrete action, trained with cross-entropy.
Actions. Packing: (x, y, z) end-effector displacement, each positive, negative or zero. Pouring: horizontal shift x and cup rotation φ. The policy runs at 10 Hz.
Tasks. Packing inserts a peg into a 3D-printed base hiding one of four bumps, or a glass into a real box holding a metal board, stuffed toy, towel and plate. Pouring tips 1 mm beads from a held cup (60 g or 100 g) to put 40 g into a cup on a scale.

Results

3D-printed packing (10 demos and 10 test trials per base): 100% success with all three senses, vs 25% vision only, 60% vision+audio, 75% vision+touch, 60% for direct feature concatenation and 48% for Du et al.’s LSTM fusion. Audio tells the hard and soft slanted bumps apart; touch tells the two flat bumps apart.
Real cluttered box: 80% success, vs 60% for concatenation and 30% for Du et al.
Pouring: mean error 1.41 g from the target, vs 2.46 g for concatenation, 5.02 g for Du et al. and 40.87 g for vision only, which pours out every bead.

Limitations

Behavior cloning needs human demonstrations, and the way the robot perceives the signals may differ from how the human demonstrator used them. The microphone is attached to the task object rather than built into a general gripper alongside the tactile sensor.

Videos

Figures

From Figure 1: The real-world dense packing task: the robot inserts a glass into a densely cluttered box using what i... — From Figure 1: The real-world dense packing task: the robot inserts a glass into a densely cluttered box using what it sees, hears and feels.

From Figure 2: Camera, contact-microphone and tactile data (left) are fused by the multisensory self-attention model ... — From Figure 2: Camera, contact-microphone and tactile data (left) are fused by the multisensory self-attention model (middle) to predict actions for dense packing and pouring (right).

From Figure 4: Dense packing setup: a 3D-printed walled base with one of four bumps inside: hard slanted, soft slante... — From Figure 4: Dense packing setup: a 3D-printed walled base with one of four bumps inside: hard slanted, soft slanted, left flat or back flat.

From Figure 5: Attention per modality over one dense packing trial: no contact before insertion, informative sound an... — From Figure 5: Attention per modality over one dense packing trial: no contact before insertion, informative sound and touch when the peg hits the bump, and a plain push down once it has passed.

Abstract

Humans use all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies on one, or occasionally two modalities, such as vision and touch. In this work, we systematically study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks. We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor, with all three sensory modalities fused with a self-attention model. Results on two challenging tasks, dense packing and pouring, demonstrate the necessity and power of multisensory perception for robotic manipulation: vision displays the global status of the robot but can often suffer from occlusion, audio provides immediate feedback of key moments that are even invisible, and touch offers precise local geometry for decision making. Leveraging all three modalities, our robotic system significantly outperforms prior methods.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong