Deep Dynamics Models for Learning Dexterous Manipulation

Anusha Nagabandi, Kurt Konoglie, Sergey Levine, Vikash Kumar

Seminal September 2019

TL;DR PDDM learns an ensemble of neural-network dynamics models and plans through them with a smoothed, reward-weighted MPC optimizer. On a real 24-DoF Shadow Hand it learns to rotate two Baoding balls from 4 hours of purely real-world data, with about 100% success on 90° turns.

What they do

PDDM (online planning with deep dynamics models) applies model-based RL to dexterous manipulation. It learns a neural-network model of the hand and objects and plans through it with MPC. The tasks are valve turning with a 9-DoF three-fingered hand, plus cube reorientation, handwriting and Baoding-ball rotation with a 24-DoF hand. The Baoding task also runs on a real Shadow Hand, trained only on real-world data.

How it works

Dynamics model. An ensemble of fully connected networks (2 hidden layers of 500) predicts the change in state. Each member starts from a different random initialization and trains on different batches, which gives an estimate of model uncertainty.
Planner. At each step the planner samples N action sequences over a short horizon (7 steps) and scores them by the reward averaged over the ensemble. It updates the sampling mean with soft reward weighting (factor γ). The noise is filtered so actions stay smooth over time, which also shrinks the search space.
Loop. The robot executes only the first action and then replans. Collected data is added to the dataset and the model is retrained. Because the model is separate from the reward, the reward or goal can be swapped at run time.

Results

Ablations on Baoding balls: model capacity and ensembles matter. PDDM’s controller greatly outperforms CEM and random shooting, and medium γ values work best.
On handwriting of arbitrary trajectories and on simulated Baoding balls, only PDDM succeeds. Baoding takes 100,000 data points (2.7 hours of data). With 8 reorientation goals, the model-free baselines get stuck in local optima.
The simulated Baoding model can be reused without retraining for related tasks, such as moving one ball to a goal or rotating clockwise.
On the real Shadow Hand (stereo-camera ball tracker, automated reset with a Franka arm), success is about 100% on 90° turns and about 54% on 180° turns, from 4 hours of data.

Limitations

The paper leaves sparse-reward and long-horizon tasks to future work, along with adding vision and touch to the model. All rewards are hand-designed and dense. The real-world setup depends on an external ball tracker and a reset mechanism.

Videos

Figures

From Figure 1: Using less than 4 hours of experience, the learned model lets the Shadow Hand rotate two free-floating... — From Figure 1: Using less than 4 hours of experience, the learned model lets the Shadow Hand rotate two free-floating Baoding balls in its palm, with no prior knowledge of the system dynamics.

From Figure 10: Overview of PDDM. A neural-network dynamics model is retrained every so often on the stored trajector... — From Figure 10: Overview of PDDM. A neural-network dynamics model is retrained every so often on the stored trajectories, and at run time a controller uses its predictions and the reward function to choose the commands sent to the hand.

From Figure 8: Simulated Baoding balls task. PDDM outperforms the prior model-based methods (MBPO, PETS, Nagabandi et... — From Figure 8: Simulated Baoding balls task. PDDM outperforms the prior model-based methods (MBPO, PETS, Nagabandi et al.) and model-free methods (SAC, NPG).

From Figure 9: Real-world Baoding balls on the Shadow Hand. Success rates for 90 and 180 degree turns against hours o... — From Figure 9: Real-world Baoding balls on the Shadow Hand. Success rates for 90 and 180 degree turns against hours of real-world training data.

Abstract

Dexterous multi-fingered hands can provide robots with the ability to flexibly perform a wide range of manipulation skills. However, many of the more complex behaviors are also notoriously difficult to control: Performing in-hand object manipulation, executing finger gaits to move objects, and exhibiting precise fine motor skills such as writing, all require finely balancing contact forces, breaking and reestablishing contacts repeatedly, and maintaining control of unactuated objects. Learning-based techniques provide the appealing possibility of acquiring these skills directly from data, but current learning approaches either require large amounts of data and produce task-specific policies, or they have not yet been shown to scale up to more complex and realistic tasks requiring fine motor skills. In this work, we demonstrate that our method of online planning with deep dynamics models (PDDM) addresses both of these limitations; we show that improvements in learned dynamics models, together with improvements in online model-predictive control, can indeed enable efficient and effective learning of flexible contact-rich dexterous manipulation skills -- and that too, on a 24-DoF anthropomorphic hand in the real world, using just 4 hours of purely real-world data to learn to simultaneously coordinate multiple free-floating objects. Videos can be found at https://sites.google.com/view/pddm/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong