Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning

Yuanpei Chen, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Stephen Marcus McAleer, Yiran Geng, Hao Dong, Zongqing Lu, Song-Chun Zhu, Yaodong Yang

Seminal June 2022

TL;DR Bi-DexHands is an Isaac Gym benchmark of 20 two-Shadow-Hand tasks matched to the ages at which children learn them, running at 30,000+ FPS on one RTX 3090. PPO solves most single tasks, but multi-task PPO averages 8.9 reward over all 20 tasks against 32.5 for per-task PPO, and meta RL barely beats a random policy.

What they do

Bi-DexHands is a simulated benchmark for bimanual dexterous manipulation with RL. Two Shadow Hands perform 20 tasks such as handing over, catching, opening bottle caps, lifting pots and stacking blocks. Each task is mapped to the age at which children acquire the skill, using the Fine Motor Subtest and other cognitive-science sources, from 5-6 months (push block) up to 48 months (catch, pour water), plus one adult-level task (two simultaneous catches). The paper then benchmarks single-agent, multi-agent, offline, multi-task and meta RL on these tasks.

How it works

Hands. Each Shadow Hand has 24 DoF, 20 of them actuated. The policy outputs relative joint targets at 30 Hz to a 1 kHz controller, and in some tasks it also moves the hand base. Observations are 398-446-dimensional states; actions have 40-52 dimensions.
Objects and rewards. Objects come from YCB and SAPIEN. Rewards share one structure: each hand’s distance to its contact point plus the object’s distance to its goal.
Settings. In the multi-agent setting each hand (or finger) is an agent with a shared reward. The multi-task and meta splits (MT1/ML1, MT4/ML4, MT20/ML20) follow Meta-World.

Results

PPO runs at 32,000-36,000 FPS and HAPPO at about 23,000 FPS on one RTX 3090.
PPO does well on most tasks and beats MAPPO and HAPPO in most cases. The gap narrows on harder two-hand tasks. SAC fails on almost all tasks; on the Humanoid task, SAC beats PPO below 128 environments but not above.
Offline RL on Hand Over medium-expert data: TD3+BC scores 84.9 and BCQ 81.7, against 63.3 for behavior cloning.
Generalization: multi-task PPO averages 8.9 reward on MT20 against 32.5 for per-task PPO. On the ML20 test tasks, ProMP gets 0.36 against 0.27 for a random policy. Tasks tied to older ages are generally harder for multi-task PPO.

Limitations

No deformable objects, and no real-robot experiments. Policies use state observations, which the authors note makes sim-to-real transfer difficult. Point-cloud input runs at only 200+ FPS with at most 256 environments, so it is not part of the baselines.

Figures

From Figure 2: The Bi-DexHands framework. Tasks built from atomic actions and a dataset of hands and objects form a d... — From Figure 2: The Bi-DexHands framework. Tasks built from atomic actions and a dataset of hands and objects form a distribution of parallel environments, used by single-agent, multi-agent, multi-task, meta and offline RL algorithms.

From Figure 5: All 20 tasks, from opening a pen cap and pressing switches to catching thrown objects and opening doors.

From Figure 3: Learning curves of the RL and multi-agent RL baselines on all 20 tasks, over 10 trials, 100M steps and... — From Figure 3: Learning curves of the RL and multi-agent RL baselines on all 20 tasks, over 10 trials, 100M steps and 2048 parallel environments.

From Figure 4: Normalized reward of multi-task PPO on each MT20 task (0 is a random policy, 1 is single-task PPO), wi... — From Figure 4: Normalized reward of multi-task PPO on each MT20 task (0 is a random policy, 1 is single-task PPO), with tasks ordered left to right by the human age they correspond to.

Abstract

Achieving human-level dexterity is an important open problem in robotics. However, tasks of dexterous hand manipulation, even at the baby level, are challenging to solve through reinforcement learning (RL). The difficulty lies in the high degrees of freedom and the required cooperation among heterogeneous agents (e.g., joints of fingers). In this study, we propose the Bimanual Dexterous Hands Benchmark (Bi-DexHands), a simulator that involves two dexterous hands with tens of bimanual manipulation tasks and thousands of target objects. Specifically, tasks in Bi-DexHands are designed to match different levels of human motor skills according to cognitive science literature. We built Bi-DexHands in the Issac Gym; this enables highly efficient RL training, reaching 30,000+ FPS by only one single NVIDIA RTX 3090. We provide a comprehensive benchmark for popular RL algorithms under different settings; this includes Single-agent/Multi-agent RL, Offline RL, Multi-task RL, and Meta RL. Our results show that the PPO type of on-policy algorithms can master simple manipulation tasks that are equivalent up to 48-month human babies (e.g., catching a flying object, opening a bottle), while multi-agent RL can further help to master manipulations that require skilled bimanual cooperation (e.g., lifting a pot, stacking blocks). Despite the success on each single task, when it comes to acquiring multiple manipulation skills, existing RL algorithms fail to work in most of the multi-task and the few-shot learning settings, which calls for more substantial development from the RL community. Our project is open sourced at https://github.com/PKU-MARL/DexterousHands.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong