Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes

Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, Pulkit Agrawal

Seminal November 2022

Teaser showing the hand turning an object to a displayed goal orientation.

TL;DR One RL policy trained in simulation uses a single depth camera to reorient new, complex objects to any orientation with a downward-facing, low-cost open-source hand, median time under 7 s. On a table, 55% of trials on unseen shapes end within 0.4 rad; in the air, an unseen duck is dropped 56% of the time.

What they do

In-hand reorientation turns a held object from any start orientation to any target in the full rotation space. Here the hand faces downward, so the fingers must also counter gravity, and the only sensors are one RealSense depth camera and the joint encoders. One controller, trained in simulation on 150 objects, runs on the open-source three-fingered D’Claw and a four-fingered version.

How it works

Teacher. PPO in Isaac Gym with privileged state (object pose and velocity, fingertip states) but no object shape. A penalty on object-table contact makes the four-fingered hand learn in-air reorientation.
Two-stage student. A sparse 3D CNN with a GRU, trained with DAgger, maps the camera, hand and goal-object point clouds to actions at 12 Hz. Stage 1 uses unoccluded point clouds sampled from meshes, skipping rendering: under three days instead of 20+. Stage 2 finetunes on rendered depth in about one day.
Sim-to-real. CMA-ES over parallel simulations identifies the joint dynamics, so only objects get wide randomization. Fingertips are soft elastomer.
Stopping. A head predicts the remaining rotation; the robot stops when it and the action are small.

Results

On a table with soft fingertips, 79% of trials on 7 training shapes and 55% on 5 unseen shapes end within 0.4 rad (rigid fingertips: 81% and 45%). Within 0.8 rad: 98% and 86%.
Error is similar across five table surfaces, including slippery acrylic and a perforated bath mat.
In the air, median reorientation time is under 7 s. A training shape (truck) drops in 23% of trials and an unseen duck in 56% (about 20% in simulation). Undropped trials have errors similar to the table-top case. Only the four-fingered hand learned in-air reorientation.

Limitations

Precision and dropping are the weak points. Most high-error runs come from stopping at the wrong moment; nearly symmetric object #12 often stops about 180° off. The authors suspect 12 Hz visual feedback and the lack of touch sensing cause drops, but do not test this. Training objects are all asymmetric or only reflection-symmetric.

Videos

In-air reorientation through a sequence of goal orientations, stopping at each.

The hand, mounted on an arm, takes an object from a person and reorients it.

Figures

From Figure 1: The real-world setup, where a neural network maps one depth camera and the joint positions to changes ... — From Figure 1: The real-world setup, where a neural network maps one depth camera and the joint positions to changes in joint positions, and the same controller reorienting three objects. The rightmost column shows the target orientation; the first two rows are in the air with a four-fingered hand, the last uses a supporting surface.

From Figure 6: A teacher policy is trained with RL on privileged state. A student learns to imitate it from synthetic... — From Figure 6: A teacher policy is trained with RL on privileged state. A student learns to imitate it from synthetic, complete point clouds, is finetuned on rendered point clouds, and then controls the real robot directly.

From Figure 4: Three and four fingers learn equally well with a supporting surface, but when rewarded for lifting the... — From Figure 4: Three and four fingers learn equally well with a supporting surface, but when rewarded for lifting the object the four-fingered hand learns much better. Below: real in-air error and episode time for one training and one unseen object, and on-table error for twelve objects and two symmetric objects.

Abstract

In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in less structured environments that remain beyond the reach of current robots. Prior works built reorientation systems assuming one or many of the following: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasistatic manipulation, simulation-only results, the need for specialized and costly sensor suites, and other constraints which make the system infeasible for real-world deployment. We present a general object reorientation controller that does not make these assumptions. It uses readings from a single commodity depth camera to dynamically reorient complex and new object shapes by any rotation in real-time, with the median reorientation time being close to seven seconds. The controller is trained using reinforcement learning in simulation and evaluated in the real world on new object shapes not used for training, including the most challenging scenario of reorienting objects held in the air by a downward-facing hand that must counteract gravity during reorientation. Our hardware platform only uses open-source components that cost less than five thousand dollars. Although we demonstrate the ability to overcome assumptions in prior work, there is ample scope for improving absolute performance. For instance, the challenging duck-shaped object not used for training was dropped in 56 percent of the trials. When it was not dropped, our controller reoriented the object within 0.4 radians (23 degrees) 75 percent of the time. Videos are available at: https://taochenshh.github.io/projects/visual-dexterity.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong