Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations
TL;DR Model-free RL learns four dexterous tasks on a simulated 24-DoF hand, and DAPG adds 25 VR demonstrations per task to policy gradient. With only a sparse success reward it reaches 90% success in 3-6 robot hours, against 16-98 hours for RL from scratch with shaped rewards.
What they do
The paper defines four MuJoCo tasks for the 24-DoF ADROIT hand: move a ball to a target, reorient a pen in the hand, unlatch and open a door, and hammer in a nail. Natural policy gradient (NPG) can solve them from scratch, but it needs hand-tuned shaped rewards and many samples, and the policies it learns move erratically and are brittle. DAPG (Demo Augmented Policy Gradient) adds a small set of demonstrations to policy gradient, so the tasks can be learned from a sparse task-completion reward alone.
How it works
- Demonstrations. They record 25 successful demonstrations per task in VR, using a CyberGlove III and an HTC Vive tracker. The pen task has no VR tactile feedback, so its demonstrations come from an RL expert trained on a shaped reward instead.
- Behavior cloning. The policy is first fit to the demonstrations by maximum likelihood. The cloned policies usually fail on their own, but they give exploration a good place to start.
- Augmented NPG. During fine-tuning, the NPG gradient gets an extra term on the demonstration state-action pairs. Its weight decays every iteration (lambda0 = 0.1, lambda1 = 0.95), so the demonstrations steer early learning and then fade out.
Results
- With sparse rewards, RL from scratch never succeeds except on the pen. With shaped rewards NPG reaches high success, while DDPG does not learn.
- Robot hours to reach 90% success: DAPG needs 5.8 (relocation), 6.1 (hammer), 4.7 (door) and 3.3 (pen). Shaped-reward NPG needs 98, 50, 16.2 and 96. Sparse-reward NPG learns only the pen, after 322 h.
- DAPG beats DDPGfD on all four tasks, well before DDPGfD shows any progress.
- When object mass and size change, DAPG relocation policies hold up better than shaped-reward ones. Trained on a mix of object sizes and masses, DAPG still learns, while shaped-reward RL does not.
Limitations
All results are in simulation; there are no hardware experiments. Policies see the ground-truth object pose and no touch sensing. The pen demonstrations come from an RL expert, and human-likeness is shown only in video, not measured.
Videos
Figures
Abstract
Dexterous multi-fingered hands are extremely versatile and provide a generic way to perform a multitude of tasks in human-centric environments. However, effectively controlling them remains challenging due to their high dimensionality and large number of potential contacts. Deep reinforcement learning (DRL) provides a model-agnostic approach to control complex dynamical systems, but has not been shown to scale to high-dimensional dexterous manipulation. Furthermore, deployment of DRL on physical systems remains challenging due to sample inefficiency. Consequently, the success of DRL in robotics has thus far been limited to simpler manipulators and tasks. In this work, we show that model-free DRL can effectively scale up to complex manipulation tasks with a high-dimensional 24-DoF hand, and solve them from scratch in simulated experiments. Furthermore, with the use of a small number of human demonstrations, the sample complexity can be significantly reduced, which enables learning with sample sizes equivalent to a few hours of robot experience. The use of demonstrations result in policies that exhibit very natural movements and, surprisingly, are also substantially more robust.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.