Rotating without Seeing: Towards In-hand Dexterity through Touch

Zhao-Heng Yin, Binghao Huang, Yuzhe Qin, Qifeng Chen, Xiaolong Wang

Seminal March 2023

Grid of real-hand rotation trials across many different objects.

TL;DR Covers one side of an Allegro hand with 16 cheap binary touch sensors and trains an RL policy in simulation that rotates objects in the palm from touch and joint angles alone. On the real hand it turns a cuboid 4.91 rounds in 30 s versus 0.25 without touch, and rotates unseen objects such as an orange and a soup can.

What they do

Touch Dexterity rotates an object in the palm of a multi-finger hand without vision. Instead of a few high-resolution fingertip sensors, it covers the palm, finger links and fingertips of an Allegro hand with 16 force-sensing resistors (about $12 each), each read as touch/no-touch. An RL policy trained in IsaacGym transfers directly to the real hand.

How it works

Binary touch. Real readings are thresholded; in simulation each sensor is a fixed link whose net contact force is thresholded at 0.01 N. This avoids matching noisy, nonlinear force values between sim and real; calibration is just picking the threshold.
Policy. PPO with an MLP. Input: joint positions, the 16 touch bits, the previous joint target and the rotation axis, stacked over 4 steps. Output: relative joint targets, smoothed by an exponential moving average and tracked by a PD controller at 10 Hz. Only the value network sees contact forces, object pose and physics parameters.
Reward and sim2real. Rotation is a finite-difference angle around the axis (simulator angular velocity was too noisy), plus penalties on velocity, falling, work and torque. Mass, friction, PD gains and sensor dropout and delay are randomized; PD gains are matched by system identification.

Results

In simulation, multi-object training fails completely without touch or with low-sensitivity (0.2 N) sensors. Disabling touch at test time drops cumulative rotation reward from 976 to 352 on seen objects and from 594 to 187 on unseen ones.
Real hand, 30 s trials, 3 seeds: the cuboid turns 4.91 rounds and never falls (no-sensor policy: 0.25). Unseen objects: apple 2.67, orange 3.00, soup can 4.25, rubber duck 1.42 rounds. Continuous-force input does worse and varies widely across objects.
Removing the fingertip or the palm sensors cuts cuboid rotation to 0.17 or 0.42 rounds.
From policy rollouts, a shape predictor reconstructs unseen column-shaped objects with MSE 0.22 using touch, 0.45 without.

Limitations

The task is rotation about a fixed axis in the palm, a simplified form of in-hand reorientation. Rotation about x and y is weaker (1.68 and 1.88 rounds on seen objects, versus 3.43 about z); the authors attribute this to contacts at the sides of the finger links, which the sensor layout does not cover.

Videos

The real Allegro hand rotating a red object about the x-axis by touch alone.

Figures

From Figure 1: Left, the Allegro hand with 16 force-sensing resistors, in simulation and real. Right, a policy traine... — From Figure 1: Left, the Allegro hand with 16 force-sensing resistors, in simulation and real. Right, a policy trained in simulation to rotate objects around different axes, running on the real hand and rotating unseen test objects.

From Figure 3: The control loop. Stacked joint positions, contact signals and previous targets go into the policy; it... — From Figure 3: The control loop. Stacked joint positions, contact signals and previous targets go into the policy; its relative action gives the next joint target, which a PD controller tracks.

From Figure 2: The two things the sensors provide: where the object sits in the hand, and whether the finger doing th... — From Figure 2: The two things the sensors provide: where the object sits in the hand, and whether the finger doing the rotation is actually in contact.

From Figure 6: Training curves with and without touch sensors for single-object and multi-object training, averaged o... — From Figure 6: Training curves with and without touch sensors for single-object and multi-object training, averaged over 3 seeds; shaded areas show the standard deviation.

Abstract

Tactile information plays a critical role in human dexterity. It reveals useful contact information that may not be inferred directly from vision. In fact, humans can even perform in-hand dexterous manipulation without using vision. Can we enable the same ability for the multi-finger robot hand? In this paper, we present Touch Dexterity, a new system that can perform in-hand object rotation using only touching without seeing the object. Instead of relying on precise tactile sensing in a small region, we introduce a new system design using dense binary force sensors (touch or no touch) overlaying one side of the whole robot hand (palm, finger links, fingertips). Such a design is low-cost, giving a larger coverage of the object, and minimizing the Sim2Real gap at the same time. We train an in-hand rotation policy using Reinforcement Learning on diverse objects in simulation. Relying on touch-only sensing, we can directly deploy the policy in a real robot hand and rotate novel objects that are not presented in training. Extensive ablations are performed on how tactile information help in-hand manipulation.Our project is available at https://touchdexterity.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong