Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment

Huihan Liu, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, Yuke Zhu

Seminal November 2022

Sirius on real Gear Insertion, inserting precisely and recovering from failed attempts.

TL;DR Sirius deploys a learned manipulation policy with a human who takes over when it is about to fail, then retrains the policy on that deployment data, upweighting interventions and discarding the robot steps just before them. Autonomous success beats state-of-the-art baselines by 8% in simulation and 27% on a real robot.

What they do

Learned manipulation policies are not reliable enough to deploy on their own. Sirius deploys one anyway, with a human supervisor. The robot acts by default; the human watches, takes over with a 6-DoF SpaceMouse when the robot is about to fail, and hands control back once the robot is in a safe state. This team completes every task, and the data it produces retrains the policy in rounds so that the human is needed less over time.

How it works

Rounds. A BC-RNN policy is warm-started on 30-80 demonstrations. It takes workspace and eye-in-hand images plus proprioception and outputs a GMM over actions. In each round the current policy is deployed while the next one trains on all data collected so far.
Intervention-guided weights. Each sample is labelled demo, robot, intervention or pre-intervention. Pre-intervention means the 15 steps before a takeover, about the operator’s 2 s reaction time. Weighted behavioral cloning resamples the classes to a target mix: interventions at 50%, pre-interventions at 0, demos at their true share and robot samples making up the rest.
Memory management. With a fixed-size buffer they compare five eviction rules. LFI evicts the trajectories with the fewest interventions first.

Results

Tasks: Nut Assembly and Tool Hang in simulation; Gear Insertion and Coffee Pod Packing on a real Franka. After three rounds Sirius has the highest autonomous success on all four. It beats IWR, as well as BC-RNN and IQL trained on the same deployment data. The reported gain over state-of-the-art methods is 8% in simulation and 27% on hardware.
Removing any of the demo, intervention or pre-intervention weights lowers success, and performance peaks at an intervention share of 0.5.
With memory capped at 500 trajectories (15% of Nut Assembly’s 3000+, 30% of Tool Hang’s 1600+), LFI matches or beats keeping all the data and converges twice as fast.
The share of intervention samples falls every round, and falls faster than with IWR.

Limitations

Each task used a single operator, so results may reflect that person’s skill and risk tolerance. The human still has to monitor the robot constantly. Workload is measured only as the fraction of intervention samples, not with a human study.

Videos

IWR baseline on Gear Insertion, failing its reattempts and getting stuck at insertion.

IWR baseline on Coffee Pod Packing, with imprecise insertion and a wrong release state.

Figures

From Figure 1: Sirius overview. The robot runs its policy autonomously while a human monitors and takes over through ... — From Figure 1: Sirius overview. The robot runs its policy autonomously while a human monitors and takes over through teleoperation when needed; deployment data go to memory and train the next policy.

From Figure 3: Deployment data fall into four classes: human demonstrations, robot actions, human interventions and p... — From Figure 3: Deployment data fall into four classes: human demonstrations, robot actions, human interventions and pre-interventions (the steps just before an intervention). Samples are reweighted by class, with interventions weighted highest, and the policy is trained with weighted behavioral cloning.

From Figure 5: Autonomous success rate over three rounds on Nut Assembly, Tool Hang, Gear Insertion (real) and Coffee... — From Figure 5: Autonomous success rate over three rounds on Nut Assembly, Tool Hang, Gear Insertion (real) and Coffee Pod Packing (real). Sirius (Ours) ends highest on all four, ahead of IWR and of BC-RNN and IQL trained on Sirius deployment data.

From Figure 9: Robot actions (yellow) and human interventions (green) over 10 consecutive Gear Insertion trials. Roun... — From Figure 9: Robot actions (yellow) and human interventions (green) over 10 consecutive Gear Insertion trials. Round 0 needs many interventions; in Round 3 the robot runs autonomously most of the time.

Abstract

With the rapid growth of computing powers and recent advances in deep learning, we have witnessed impressive demonstrations of novel robot capabilities in research settings. Nonetheless, these learning systems exhibit brittle generalization and require excessive training data for practical tasks. To harness the capabilities of state-of-the-art robot learning models while embracing their imperfections, we present Sirius, a principled framework for humans and robots to collaborate through a division of work. In this framework, partially autonomous robots are tasked with handling a major portion of decision-making where they work reliably; meanwhile, human operators monitor the process and intervene in challenging situations. Such a human-robot team ensures safe deployments in complex tasks. Further, we introduce a new learning algorithm to improve the policy's performance on the data collected from the task executions. The core idea is re-weighing training samples with approximated human trust and optimizing the policies with weighted behavioral cloning. We evaluate Sirius in simulation and on real hardware, showing that Sirius consistently outperforms baselines over a collection of contact-rich manipulation tasks, achieving an 8% boost in simulation and 27% on real hardware than the state-of-the-art methods in policy success rate, with twice faster convergence and 85% memory size reduction. Videos and more details are available at https://ut-austin-rpl.github.io/sirius/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong