3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing

Binghao Huang, Yixuan Wang, Xinyi Yang, Yiyue Luo, Yunzhu Li

Seminal October 2024

Egg steaming policy rollout beside its visuo-tactile point cloud, 2x speed.

TL;DR Thin piezoresistive tactile pads (16x16 per finger) on soft grippers, with tactile readings placed as points next to the camera point cloud and fed to a diffusion policy. Across four bimanual tasks it reaches 80-90% whole-task success, against 45-65% for point-cloud-only policies.

What they do

Bimanual manipulation of fragile objects or tools held in the hand needs contact information that cameras miss under occlusion. 3D-ViTac adds dense, flexible tactile pads to soft fin-shaped grippers on a low-cost bimanual teleoperation rig. Touch and vision go into one 3D point cloud for a diffusion policy trained by imitation.

How it works

Sensor. Each finger carries a 16x16 piezoresistive array (Velostat between orthogonal conductive threads), under 1 mm thick, with 3 mm² per sensing point. It reads out at up to about 32.2 FPS through an Arduino, and a pad plus reading board costs about $20, excluding the Arduino. Four pads give 1024 units.
Representation. Three RealSense views are merged, cropped and downsampled to 512 visual points. Each tactile unit becomes a 3D point, placed by forward kinematics, carrying its continuous reading as a feature. A one-hot flag marks each point as visual or tactile.
Policy. A PointNet++ encoder conditions a diffusion policy that outputs sequences of robot joint states. During teleoperation, tactile readings are shown to the operator on a screen.

Results

Whole-task success over 20 episodes, from 30-50 demos: egg steaming 0.85, fruit preparation 0.80, hex key collection 0.90, sandwich serving 0.85. Point-cloud-only policies score 0.55, 0.45, 0.65 and 0.65. The best tactile-image baselines reach 0.70, 0.70, 0.60 and 0.75.
With a single camera, adding tactile points raises success from 0.50 to 0.80 (egg steaming) and from 0.30 to 0.70 (hex key). Tactile points alone reach only 0.10 and 0.15.
Ablations: 16x16 resolution slightly beats 8x8 and clearly beats 4x4 and binary contact.
Single-arm peg insertion: 0.85 versus 0.65 for point cloud only.

Limitations

Collecting real multimodal demonstrations is expensive. The tactile sensor has no simulation model, so the data cannot be randomized or scaled up in simulation.

Videos

Fruit preparation, picking grapes out of a plastic bag onto a plate, 2x speed.

Hex key collection, with a handover and in-hand adjustment before insertion, 2x speed.

Figures

From Figure 1: The 3D-ViTac bimanual system, with a 16x16 tactile array on each finger, and the four tasks that use f... — From Figure 1: The 3D-ViTac bimanual system, with a 16x16 tactile array on each finger, and the four tasks that use force and in-hand position information from touch.

From Figure 2: (a) The bimanual setup with four tactile pads on soft grippers and tactile readings shown on a screen;... — From Figure 2: (a) The bimanual setup with four tactile pads on soft grippers and tactile readings shown on a screen; (b) the tactile soft gripper, 256 sensing units per pad, and the readout board; (c) sensor response to force and consistency across the pad.

From Figure 3: (a) Real-world setup; (b) visual points and tactile points, placed by robot kinematics, merged in one ... — From Figure 3: (a) Real-world setup; (b) visual points and tactile points, placed by robot kinematics, merged in one 3D frame; (c) the policy takes the visuo-tactile points and outputs an action sequence.

From Figure 13: Typical baseline failures on the four tasks, such as dropped eggs, crushed grapes, a slipping hex key... — From Figure 13: Typical baseline failures on the four tasks, such as dropped eggs, crushed grapes, a slipping hex key and a misplaced fried egg.

Abstract

Tactile and visual perception are both crucial for humans to perform fine-grained interactions with their environment. Developing similar multi-modal sensing capabilities for robots can significantly enhance and expand their manipulation skills. This paper introduces \textbf{3D-ViTac}, a multi-modal sensing and learning system designed for dexterous bimanual manipulation. Our system features tactile sensors equipped with dense sensing units, each covering an area of 3$mm^2$. These sensors are low-cost and flexible, providing detailed and extensive coverage of physical contacts, effectively complementing visual information. To integrate tactile and visual data, we fuse them into a unified 3D representation space that preserves their 3D structures and spatial relationships. The multi-modal representation can then be coupled with diffusion policies for imitation learning. Through concrete hardware experiments, we demonstrate that even low-cost robots can perform precise manipulations and significantly outperform vision-only policies, particularly in safe interactions with fragile items and executing long-horizon tasks involving in-hand manipulation. Our project page is available at \url{https://binghao-huang.github.io/3D-ViTac/}.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong