Learning Visuotactile Skills with Two Multifingered Hands

Toru Lin, Yu Zhang, Qiyang Li, Haozhi Qi, Brent Yi, Sergey Levine, Jitendra Malik

Seminal April 2024

Steak Serving: one hand holds the pan, the other lifts the steak onto the plate with a spatula.

TL;DR HATO is a low-cost VR teleoperation system for two UR5e arms fitted with touch-sensing prosthetic hands, used to train diffusion policies on vision, touch and proprioception. The policies succeed 9-10/10 on handover, stacking and pouring and 5/10 on steak serving, which drops to 0/10 without touch.

What they do

Instead of the usual parallel-jaw grippers, this paper fits two UR5e arms with Psyonic Ability Hands, prosthetic hands repurposed for research with custom PCBs, whose fingertips carry six touch sensors each. HATO, a teleoperation system built on a Meta Quest 2, collects demonstrations that train diffusion policies on vision, touch and proprioception for four tasks: slippery handover, tower block stacking, wine pouring and steak serving.

How it works

Teleoperation. Each controller’s pose sets the arm’s end-effector target through IK. The grip button closes the four non-thumb fingers in proportion to the press; the thumbstick drives the thumb’s two joints. The trigger pauses arm control so the operator can reposition.
Data. Recorded at 10 Hz: proprioception, RGB-D from two wrist cameras and one head-view camera, and 60 touch readings across both hands.
Policy. A DDPM diffusion policy takes a single observation and predicts 16 future 24-D joint targets for both arms and hands. Each camera has its own ResNet-18; touch and end-effector poses go through small MLPs.
Deployment. A remote server runs inference asynchronously on the latest observation; overlapping predictions are averaged for smoother motion.

Results

From 100-300 demonstrations per task (30 min to 2 h of teleoperation), success over 10 trials is 10/10 handover, 10/10 stacking, 9/10 pouring and 5/10 steak serving; objects are picked up 10/10 on all tasks.
Steak serving: without vision the policy never picks up the objects; without touch it picks them up but serves 0/10, even though its action error is similar (0.07 vs 0.08).
Stacking with rotated blocks: 4/10 without touch, 0/10 without vision.
Wrist cameras give lower error than the third-view camera on three of four tasks; adding depth gives no marked benefit.

Limitations

The grip-button mapping moves the four fingers together, so finger gaiting is not possible, and the teleoperator gets no haptic feedback. Policies are trained from scratch and are sensitive to appearance changes in the scene. Steak serving succeeds only half the time.

Videos

Tower Block Stacking: both hands carry the two-block pile onto the yellow block.

Teleoperating Wine Pouring with a pair of Quest controllers.

Figures

From Figure 1: Learned policies on the four tasks: (a) Slippery Handover, (b) Tower Block Stacking, (c) Wine Pouring,... — From Figure 1: Learned policies on the four tasks: (a) Slippery Handover, (b) Tower Block Stacking, (c) Wine Pouring, where the center of mass shifts during manipulation, and (d) Steak Serving, which needs tool use and a stable grasp while moving the steak on the spatula.

From Figure 2: Six tactile sensors on each fingertip, each giving a continuous value proportional to the sensed press... — From Figure 2: Six tactile sensors on each fingertip, each giving a continuous value proportional to the sensed pressure.

From Figure 4: Held-out action prediction error with proprioception only, without vision, without touch, and with all... — From Figure 4: Held-out action prediction error with proprioception only, without vision, without touch, and with all three inputs. Removing vision raises the error on every task; removing touch changes it little on Block Stacking and Steak Serving.

Abstract

Aiming to replicate human-like dexterity, perceptual experiences, and motion patterns, we explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Two significant challenges exist: the lack of an affordable and accessible teleoperation system suitable for a dual-arm setup with multifingered hands, and the scarcity of multifingered hand hardware equipped with touch sensing. To tackle the first challenge, we develop HATO, a low-cost hands-arms teleoperation system that leverages off-the-shelf electronics, complemented with a software suite that enables efficient data collection; the comprehensive software suite also supports multimodal data processing, scalable policy learning, and smooth policy deployment. To tackle the latter challenge, we introduce a novel hardware adaptation by repurposing two prosthetic hands equipped with touch sensors for research. Using visuotactile data collected from our system, we learn skills to complete long-horizon, high-precision tasks which are difficult to achieve without multifingered dexterity and touch feedback. Furthermore, we empirically investigate the effects of dataset size, sensing modality, and visual input preprocessing on policy learning. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data. Videos, code, and datasets can be found at https://toruowo.github.io/hato/ .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong