What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martín-Martín

★ Must read August 2021

TL;DR A controlled study of six offline learning algorithms on human teleoperated demonstrations for five simulated and three real manipulation tasks. History-aware BC-RNN wins, reaching 71.3% on two-arm Transport versus 17.3% for plain BC, while offline RL methods BCQ and CQL struggle on human data.

What they do

Learning manipulation offline from human demonstrations lacked open human datasets and reproducible baselines. The authors collect teleoperated datasets for five simulated tasks (Lift, Can, Square, two-arm Transport, Tool Hang) and three real Franka tasks, train six offline algorithms on them, and distill lessons on what matters. Datasets and implementations are open-sourced.

How it works

Datasets. Proficient-Human: 200 demos from one experienced RoboTurk operator. Multi-Human: 300 demos from six operators of graded skill. Machine-Generated: rollouts of SAC checkpoints, only for Lift and Can, since RL could not solve the rest.
Algorithms. BC, BC-RNN, hierarchical BC, and the offline RL methods BCQ, CQL and IRIS, all with binary task-success rewards.
Observations. Low-dim uses end-effector pose, gripper and ground-truth object poses; image swaps the object poses for a front and a wrist camera.
Protocol. Checkpoints are evaluated with 50 rollouts during training; the best success rate is averaged over 3 seeds. Tool Hang and the real tasks get no hyperparameter tuning.

Results

BC-RNN reaches 84.0% on Square and 71.3% on Transport (Proficient-Human, low-dim), against 78.7% and 17.3% for BC. BCQ, which scores 91.3% on machine-generated Lift data, gets 7.3% on Transport and CQL 0%.
Adding velocity or joint inputs to low-dim agents costs 49-88% relative success; dropping pixel-shift randomization costs 47% on Square and 35% on Transport, dropping the wrist camera 9% and 43%.
Choosing the checkpoint by validation loss or taking the last one gives 10-100% worse policies than the best checkpoint.
Real robot (BC-RNN, 30 rollouts, final checkpoint): Lift 96.7%, Can 73.3%, Tool Hang 3.3%. Can falls to 26.7% without randomization and 43.3% without the wrist camera.

Limitations

Headline numbers rely on evaluating every checkpoint online, which the authors note is infeasible on real robots. Offline RL on human data remains unsolved, and HBC and IRIS are not tested with images. Robustness to perturbations and safety are left for future work.

Figures

From Figure 1: The simulated two-arm Transport task, where one arm fetches a hammer from a closed container while the... — From Figure 1: The simulated two-arm Transport task, where one arm fetches a hammer from a closed container while the other clears trash from the target bin before the handover.

From Figure E.2: The two observation spaces. Both include proprioception; low-dim adds ground-truth object states, im... — From Figure E.2: The two observation spaces. Both include proprioception; low-dim adds ground-truth object states, image adds an external and a wrist camera image.

From Figure 3: Low-dim BC-RNN success rate when trained on 20%, 50% and 100% of the Proficient-Human and Multi-Human ... — From Figure 3: Low-dim BC-RNN success rate when trained on 20%, 50% and 100% of the Proficient-Human and Multi-Human datasets.

From Figure 4: Relative drop in success rate when the policy is picked by lowest validation loss or as the last check... — From Figure 4: Relative drop in success rate when the policy is picked by lowest validation loss or as the last checkpoint instead of the best one, for BC and BC-RNN on Square and Transport.

Abstract

Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data. Codebase, datasets, trained models, and more available at https://arise-initiative.github.io/robomimic-web/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong