Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality

Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, Yashraj Narang, Jean-Francois Lafleche, Dieter Fox, Gavriel State

Seminal October 2022

Teaser of the real hand turning the cube to goal after goal, with a live simulator view inset.

TL;DR Train an in-hand cube reorientation policy for a low-cost Allegro Hand entirely in Isaac Gym with automatic domain randomisation, and feed it cube poses from a vision model trained only on synthetic images. Over 10 real trials the best policy averages 27.8 consecutive successes (best run 112), using three RGB cameras and no markers.

What they do

DeXtreme revisits OpenAI’s in-hand cube reorientation with cheaper parts: a four-fingered Allegro Hand with a locked wrist, three off-the-shelf RGB cameras, no markers, no cage, and GPU simulation in Isaac Gym. Both the policy and the pose estimator are trained only in simulation. A trial counts consecutive target orientations reached (within 0.4 rad) before the cube drops or gets stuck.

How it works

  • Policy. PPO trains an LSTM that sets PD targets for the 16 joints at 30 Hz. Its inputs are the noisy cube pose, the goal, joint angles and the last action. The critic also sees privileged simulator state. Training runs 16384 environments per GPU on 8 A40s.
  • Vectorised ADR. 40% of environments evaluate the policy at the edge of one randomisation range: physics, noise, latency or delay. That range widens if the policy averages over 20 successes there and shrinks if it averages under 5. Random pose injection and a random network adversary add structured noise.
  • Pose estimation. A Mask R-CNN-style network, trained on 5M randomised Isaac Sim images, predicts the cube’s 8 corner keypoints in each camera. Per-camera PnP throws out inconsistent views, and the remaining keypoints are triangulated and fit to the cube model at 15 Hz.

Results

  • Best ADR policy over 10 real trials: 27.8 ± 19.0 consecutive successes (best run 112). Repeat sessions on other days averaged 26.6 and 20.8.
  • The manual-DR policy averaged 14.8, 11.9 and 10.0, and got stuck in poses the ADR policies recovered from.
  • OpenAI’s vision policy on a different hand averaged 15.2 (best 46).
  • Training took 2.5 days, about $977 of compute against an estimated $215,685 for OpenAI’s ADR setup.

Limitations

A sim-to-real gap remains. The manual-DR policy averages 35 successes in simulation but about 15 on hardware. ADR policies also fall short of simulation and of OpenAI’s marker-based ADR (XXL) result (32). Real-world performance varies with the training seed, and the hand needs frequent repairs. ADR policies tend to cage the cube, hiding the letters the pose estimator relies on.

Videos

Uncut real-time trial counting goals reached, here past 40.
Real robot shown beside the goal cube and its simulator visualisation.
Full video (YouTube)
Full video (YouTube)

Figures

From Figure 2: The hardware setup. Unlike earlier work it is not housed in a cage and performs the task in an open la...
From Figure 2: The hardware setup. Unlike earlier work it is not housed in a cage and performs the task in an open laboratory. The background is alpha-blended for visibility.
From Figure 3: High-level overview of the system, part (a): policy training in simulation.
From Figure 3: High-level overview of the system, part (a): policy training in simulation.
From Figure 3: High-level overview of the system, part (b): the vision data generation and training pipeline.
From Figure 3: High-level overview of the system, part (b): the vision data generation and training pipeline.
From Figure 7: How the normalised parameters optimised with ADR evolve over training time, in simulation.
From Figure 7: How the normalised parameters optimised with ADR evolve over training time, in simulation.
Abstract

Recent work has demonstrated the ability of deep reinforcement learning (RL) algorithms to learn complex robotic behaviours in simulation, including in the domain of multi-fingered manipulation. However, such models can be challenging to transfer to the real world due to the gap between simulation and reality. In this paper, we present our techniques to train a) a policy that can perform robust dexterous manipulation on an anthropomorphic robot hand and b) a robust pose estimator suitable for providing reliable real-time information on the state of the object being manipulated. Our policies are trained to adapt to a wide range of conditions in simulation. Consequently, our vision-based policies significantly outperform the best vision policies in the literature on the same reorientation task and are competitive with policies that are given privileged state information via motion capture systems. Our work reaffirms the possibilities of sim-to-real transfer for dexterous manipulation in diverse kinds of hardware and simulator setups, and in our case, with the Allegro Hand and Isaac Gym GPU-based simulation. Furthermore, it opens up possibilities for researchers to achieve such results with commonly-available, affordable robot hands and cameras. Videos of the resulting policy and supplementary information, including experiments and demos, can be found at https://dextreme.org/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.