Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing

Ying Yuan, Haichuan Che, Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Kang-Won Lee, Yi Wu, Soo-Chul Lim, Xiaolong Wang

Seminal December 2023

The real hand rotates two balls around each other.

TL;DR Turn each triggered touch sensor into points on its mesh and merge them with the depth-camera point cloud, so one PointNet policy sees vision and touch in the same 3D space. Trained in simulation and transferred zero-shot to an Allegro hand, it reaches 22.9 rounds of real double-ball rotation, against 18.8 when touch is fed in as a separate vector.

What they do

In-hand rotation needs vision and touch, but touch is sparse and low-dimensional while vision is dense. Robot Synesthesia turns each triggered tactile sensor into points on that sensor’s mesh, so contact appears in the same 3D point cloud as the camera data. A policy trained in simulation runs on the real hand without real-world data on three tasks: turning a four-way wrench, rotating two balls around each other, and rotating varied objects about the x, y or z axis.

How it works

Hardware. An XArm6 arm with a 16-DoF Allegro hand, 16 force-sensing resistors (thresholded to binary contact) and an Azure Kinect depth camera, mirrored in Isaac Gym.
Tactile point cloud. Each active sensor contributes 8 points from its mesh. These join 512 camera points and 8 points per hand link sampled from the robot mesh at the current pose. A one-hot tag marks each point’s source, and all points are expressed in the palm frame.
Teacher-student. A PPO teacher sees object position, velocities and a shape embedding. It is distilled into a PointNet student by behavior cloning on 5120k transitions, then DAgger.

Results

Simulated teachers beat a proprioception-plus-touch baseline on every task (double balls: reward 1045.3 vs 620.9), while visual RL from scratch barely learns (127.8).
Simulated students: adding tactile points to camera points, robot-mesh points and the touch vector raises the wrench reward from 344.1 to 504.0 and the double-ball reward from 148.6 to 407.7. On x- and y-axis rotation it does not beat the best baseline.
Real robot, 5 trials per policy: the full model rotates the most on the wrench (1.5 vs 0.25 rounds), double balls (22.9 vs 18.8), x-axis (2.1 vs 0.7) and z-axis (10.2 vs 8.6), but not the y-axis (0.9 vs 1.4).
On average, 42.7% of the points the encoder selects are tactile points.

Limitations

The paper has no limitations section. Touch is binary contact, and every task rotates the object about a fixed axis. The authors leave goal-conditioned rotation and optical tactile sensors to future work. On the wrench, the full model drops the object sooner than the baselines (43.0 s vs 60 s).

Videos

Real-world rotation of cubes and everyday objects, shown as a grid.

Rotating a soup can, next to the point cloud the policy sees.

Figures

From Figure 1: The real-world setup, an Allegro hand with 16 force-sensing resistors and an Azure Kinect camera facin... — From Figure 1: The real-world setup, an Allegro hand with 16 force-sensing resistors and an Azure Kinect camera facing the robot.

From Figure 2: Training pipeline. An RL teacher sees proprioception, binary contact, object pose and a shape embeddin... — From Figure 2: Training pipeline. An RL teacher sees proprioception, binary contact, object pose and a shape embedding. It is distilled into a student that sees proprioception, touch and one point cloud combining camera, robot-mesh and tactile points, each tagged with a one-hot vector.

From Figure 5: Teacher learning curves on double-ball rotation, averaged over 3 seeds: rotation reward (left) and tim... — From Figure 5: Teacher learning curves on double-ball rotation, averaged over 3 seeds: rotation reward (left) and time-to-fall (right), compared with visual RL and a non-visual baseline.

From Figure 6: The points the policy's encoder selects (foreground) among all observed points (background) during eva... — From Figure 6: The points the policy's encoder selects (foreground) among all observed points (background) during evaluation. Red points come from the tactile point cloud.

Abstract

Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial information and facilitating better reasoning about robot actions. The method, trained in a simulated environment and then deployed to a real robot, is applicable to various in-hand object rotation tasks. Comprehensive ablations are performed on how the integration of vision and touch can improve reinforcement learning and Sim2Real performance. Our project page is available at https://yingyuan0414.github.io/visuotactile/ .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong