Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

Jianlan Luo, Charles Xu, Jeffrey Wu, Sergey Levine

Seminal October 2024

Motherboard assembly run autonomously at 1x speed.

TL;DR HIL-SERL trains vision-based manipulation policies with RL directly on real robots, starting from 20-30 demonstrations and taking corrections from a human with a SpaceMouse during training. It reaches 100% success on all 13 tasks, against 49.7% for an HG-DAgger baseline given the same human data, with 1.8x faster cycle times.

What they do

HIL-SERL trains vision-based manipulation policies with reinforcement learning directly on real robots, for tasks that are hard to learn from scratch: precise insertion, dual-arm assembly, a deformable timing belt, and dynamic skills such as Jenga whipping and pan flipping. The main addition over the earlier SERL system is human corrections during training.

How it works

Learning. Off-policy RL with RLPD, sampling each batch equally from a demo buffer (20-30 demonstrations) and an online RL buffer. Images are 128x128 crops from wrist and side cameras, encoded by an ImageNet-pretrained ResNet-10. A grasp critic trained with DQN picks discrete gripper actions.
Reward. A binary success classifier per task, trained on about 200 positive and 1000 negative teleoperated frames (around 5 minutes of collection, over 95% accuracy).
Human corrections. An operator takes over with a SpaceMouse when the policy gets stuck or fails. Corrections go into both buffers, the policy’s own transitions only into the RL buffer.
Control. Proprioception and actions are expressed relative to the end-effector, with randomized start poses. Contact tasks send 6D twists to an impedance controller that bounds the force; dynamic tasks command feedforward wrenches.

Results

100% success on all 13 tasks over 100 trials each (10/10 for the chained IKEA assembly), mostly within 1-2.5 h of training; the timing belt took 6 h. HG-DAgger with the same human data averages 49.7% (BC with more demonstrations for Jenga and flipping).
Average cycle time is 5.4 s against 9.6 s for the baseline.
On RAM insertion, dashboard assembly and object flipping, RL from scratch gets 0%. SERL with 200 demonstrations and no corrections averages 49%, and 0% on the dashboard. Diffusion Policy, IBRL, Residual RL and DAPG, each given 200 demonstrations, average 32-57%.
Policies recover from perturbations: a moving RAM target, a perturbed belt, a gripper forced open.

Limitations

Untested on much longer-horizon tasks, where sample complexity grows. Little randomization was used and generalization to unstructured environments was not tested. Each task is trained from scratch.

Videos

Whipping one block out of a Jenga tower, at 1x and in 8x slow motion.

Two arms grasp a panel and lift it for car dashboard assembly, 1x speed.

Figures

From Figure 1: A subset of the tasks: whipping a block out of a Jenga tower, flipping an object in a pan, and assembl... — From Figure 1: A subset of the tasks: whipping a block out of a Jenga tower, flipping an object in a pan, and assembling a timing belt, a car dashboard, a motherboard and an IKEA shelf.

From Figure 2: HIL-SERL runs an actor process and a learner process asynchronously. The actor runs the policy on the ... — From Figure 2: HIL-SERL runs an actor process and a learner process asynchronously. The actor runs the policy on the robot, where a human can take over with a SpaceMouse, and sends data to the replay buffers. The learner samples evenly from a demo buffer and an RL buffer and updates the policy with RLPD; a separate grasp critic is trained with DQN when grippers are used.

From Figure 5: Success rate, cycle time and intervention rate over training for HIL-SERL and HG-DAgger on several tas... — From Figure 5: Success rate, cycle time and intervention rate over training for HIL-SERL and HG-DAgger on several tasks (running average over 20 episodes). HIL-SERL climbs to 100% success while its intervention rate falls to 0% and cycle time drops; HG-DAgger's success and intervention rates fluctuate and its cycle time does not improve.

From Figure 7: RAM insertion, end-effector Y-Z positions over training. Rows 1-3: HIL-SERL state visits form a funnel... — From Figure 7: RAM insertion, end-effector Y-Z positions over training. Rows 1-3: HIL-SERL state visits form a funnel toward the slot, and states inside it get high Q-value variance and high Q-values. Row 4: HG-DAgger state visits are more diffuse, with a weaker funnel.

Abstract

Reinforcement learning (RL) holds great promise for enabling autonomous acquisition of complex robotic manipulation skills, but realizing this potential in real-world settings has been challenging. We present a human-in-the-loop vision-based RL system that demonstrates impressive performance on a diverse set of dexterous manipulation tasks, including dynamic manipulation, precision assembly, and dual-arm coordination. Our approach integrates demonstrations and human corrections, efficient RL algorithms, and other system-level design choices to learn policies that achieve near-perfect success rates and fast cycle times within just 1 to 2.5 hours of training. We show that our method significantly outperforms imitation learning baselines and prior RL approaches, with an average 2x improvement in success rate and 1.8x faster execution. Through extensive experiments and analysis, we provide insights into the effectiveness of our approach, demonstrating how it learns robust, adaptive policies for both reactive and predictive control strategies. Our results suggest that RL can indeed learn a wide range of complex vision-based manipulation policies directly in the real world within practical training times. We hope this work will inspire a new generation of learned robotic manipulation techniques, benefiting both industrial applications and research advancements. Videos and code are available at our project website https://hil-serl.github.io/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong