Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

General In-Hand Object Rotation with Vision and Touch

Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, Jitendra Malik

Seminal September 2023

Real-hand rotation of 24 objects, each tile labelled with its rotation axis, including diagonal axes.

TL;DR RotateIt trains an Allegro hand in simulation to rotate objects with its fingertips about the x, y or z axis, using a transformer that fuses segmented depth, fingertip contact and proprioception. Vision and touch lift x-axis rotation reward from 79 to 118 (privileged oracle 125), and the policy transfers to real objects without fine-tuning.

What they do

Earlier sim-to-real rotation policies on the Allegro hand, such as Hora, turned objects only about the z-axis. RotateIt rotates objects continuously about the x, y or z axis using only the fingertips, with no supporting surface. It is trained entirely in simulation and deployed on the real hand without fine-tuning.

How it works

  • Oracle. PPO in IsaacGym on hundreds of objects. The policy sees proprioception plus privileged information: a PointNet encoding of the object’s point cloud, and its pose and physical properties. The reward pays for spin about the target axis and penalizes spin about the other axes.
  • Visuotactile student. A transformer reads a history of segmented object depth, fingertip contact locations, joint positions and actions, and regresses the oracle’s privileged encoding.
  • Sim-to-real. Touch is reduced to one of 8 contact directions per fingertip, read on the real hand by tracking the highest-intensity pixel of a vision-based touch sensor. Real depth is masked with Segment Anything.
  • One policy is trained per axis; a distilled multi-axis policy performs on par (105 vs 110 reward on +x).

Results

  • Adding object shape lifts the oracle’s x-axis rotation reward from 85 to 125 (Hora: 79).
  • The student with vision and touch scores 118 / 109 / 136 on x / y / z, versus 79 / 67 / 108 with proprioception only. Contact location helps (102 vs 79 on x without vision); binary contact does not (80).
  • On 15 held-out harder objects, reward drops 41% with proprioception only and 15% with vision and touch.
  • Real hand, x-axis: Hora cannot finish the task and the object falls; RotateIt turns cubes, spheres and cylinders about 2π radians within 20 s, including objects outside the training set.

Limitations

Objects cannot be long (a pencil or screwdriver) and must fit the hand’s mechanical limits. The policy is frozen after training, so it cannot learn from real-world experience, and touch uses only the contact location rather than the full tactile image.

Videos

Contact on a fingertip touch sensor detected and mapped to one of 8 directions.

Figures

From Figure 1: Rotation about multiple axes on a real Allegro hand, and the inputs used: raw depth, segmented object ...
From Figure 1: Rotation about multiple axes on a real Allegro hand, and the inputs used: raw depth, segmented object depth and four fingertip tactile images.
From Figure 2: Training pipeline. The privileged encoder and control policy are trained jointly with PPO; a visuotact...
From Figure 2: Training pipeline. The privileged encoder and control policy are trained jointly with PPO; a visuotactile transformer then learns to predict the privileged encoding from a sequence of visual, tactile and proprioceptive inputs.
From Figure 8: Rotation reward on 15 held-out, harder objects. The oracle drops 22% without a point cloud and 8% with...
From Figure 8: Rotation reward on 15 held-out, harder objects. The oracle drops 22% without a point cloud and 8% with one; the sensorimotor policy drops 41% with proprioception only and 15% with vision and touch.
Abstract

We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and the importance of visual and tactile sensing.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.