Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost

Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, Vikash Kumar

Seminal October 2018

TL;DR Train neural-network policies for low-cost 3- and 4-fingered hands with model-free RL directly on the real robot, with no simulator. Valve turning, box flipping and door opening are learned in 7.4, 4.1 and 15.6 hours, and 20 human demonstrations cut valve and box training to 3.0 and 1.5 hours.

What they do

Multi-fingered hands are hard to model and control because of their many joints and intermittent contacts. This paper trains neural-network policies for them with model-free deep RL directly in the real world, without a simulator or a model. It uses two low-cost hands: the 3-finger, 9-DoF Dynamixel claw (Dclaw, under $2,500) and the 16-DoF Allegro hand (about $15,000). It also shows that a few human demonstrations speed training up.

How it works

  • Tasks. Rotate a valve 180°, flip a box 180° about its long axis, and open a door with the hand mounted on a Sawyer arm. The state is joint angles, object angle, distance to goal and last action; actions are joint position targets.
  • Instrumented objects. Dynamixel servos attached to the valve, box and door measure the object’s angle and reset it between episodes.
  • Learning. Truncated natural policy gradient from scratch. DAPG instead pretrains with behavior cloning on 20 kinesthetic demonstrations, then adds a weighted term to the policy gradient that pulls toward the demonstrated actions.
  • Design choices were picked in simulation: position control vibrates least and learns best, and a velocity penalty in the reward slows learning.

Results

  • From scratch, the deterministic policy reaches 100% success over 10 rollouts after 7.4 h (valve), 4.1 h (box) and 15.6 h (door).
  • DAPG cuts this to 3.0 h (valve) and 1.5 h (box), and its motions are gentler.
  • With the same algorithm and hyperparameters, the Allegro hand learns the valve in time comparable to the Dclaw, and the Dclaw also learns to turn a deformable foam valve.
  • In a sim-to-real comparison on the valve, randomizing PID gains and friction was crucial for transfer.

Limitations

Object state comes from servos built into each object, not from vision, and each skill is learned separately; the authors list both as future work. The door task, rewarded only once the door opens, took 15.6 h. DAPG policies were more robust to the valve’s initial angle but less robust to injected action and observation noise than policies trained from scratch.

Videos

Full video (YouTube)

Figures

From Figure 1: Opening a door with a flexible handle, with the hand mounted on a Sawyer arm.
From Figure 1: Opening a door with a flexible handle, with the hand mounted on a Sawyer arm.
From Figure 14: Kinesthetic teaching used to collect demonstrations with the Dclaw.
From Figure 14: Kinesthetic teaching used to collect demonstrations with the Dclaw.
From Figure 10: Learning progress of model-free RL from scratch. Policies reach 100% success after 7.4 hours for valv...
From Figure 10: Learning progress of model-free RL from scratch. Policies reach 100% success after 7.4 hours for valve rotation, 4.1 hours for box flipping and 15.6 hours for door opening.
From Figure 15: Valve rotation learning progress from scratch versus with DAPG, normalized so a random policy scores ...
From Figure 15: Valve rotation learning progress from scratch versus with DAPG, normalized so a random policy scores 0 and the best DAPG policy scores 1.
Abstract

Dexterous multi-fingered robotic hands can perform a wide range of manipulation skills, making them an appealing component for general-purpose robotic manipulators. However, such hands pose a major challenge for autonomous control, due to the high dimensionality of their configuration space and complex intermittent contact interactions. In this work, we propose deep reinforcement learning (deep RL) as a scalable solution for learning complex, contact rich behaviors with multi-fingered hands. Deep RL provides an end-to-end approach to directly map sensor readings to actions, without the need for task specific models or policy classes. We show that contact-rich manipulation behavior with multi-fingered hands can be learned by directly training with model-free deep RL algorithms in the real world, with minimal additional assumption and without the aid of simulation. We learn a variety of complex behaviors on two different low-cost hardware platforms. We show that each task can be learned entirely from scratch, and further study how the learning process can be further accelerated by using a small number of human demonstrations to bootstrap learning. Our experiments demonstrate that complex multi-fingered manipulation skills can be learned in the real world in about 4-7 hours for most tasks, and that demonstrations can decrease this to 2-3 hours, indicating that direct deep RL training in the real world is a viable and practical alternative to simulation and model-based control. https://sites.google.com/view/deeprl-handmanipulation

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.