QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
TL;DR Off-policy Q-learning on 580k real grasp attempts from 7 robots, choosing actions by optimizing the Q-function instead of training an actor. The closed-loop, RGB-only policy grasps 96% of unseen objects, against 78% for a prior self-supervised grasping method.
What they do
Most learned grasping systems choose a grasp pose and then execute it open-loop. QT-Opt treats grasping as closed-loop reinforcement learning instead: at every step a Q-function scores gripper motions from one over-the-shoulder RGB image, and the only reward is whether an object is lifted at the end. The training data is 580k grasp attempts from 7 KUKA LBR IIWA arms over about 800 robot hours, labeled automatically.
How it works
- Actor-free Q-learning. The cross-entropy method (64 samples, best 6, two iterations) maximizes the Q-function over continuous actions, both to compute training targets and to act on the robot.
- State, action, reward. A 472x472 image plus gripper open/closed status and height. Actions are Cartesian translation, rotation about the vertical axis, gripper open/close and a learned terminate action. Reward is 1 for a lift, with -0.05 per step.
- Stability. Cross-entropy loss, a Polyak-averaged target network and clipped double Q-learning.
- Distributed training. Replay buffers merge offline logs with live robot data, 1000 Bellman updater jobs compute targets asynchronously, and 10 GPUs train. Training starts fully off-policy, then jointly finetunes with on-policy data.
Results
- Unseen test objects (102 grasps per robot): 96% after joint finetuning with 28k on-policy grasps and 87% fully off-policy, against 78% for Levine et al. trained on 900k grasps.
- Bin emptying (28 objects, 30 grasps): 88%, 88% and 76% over the first 10, 20 and 30 grasps, against 76%, 72% and 72%.
- Singulating an unseen blocks puzzle succeeds 19/24 (baseline 13/24); grasping a tennis ball pushed out of the gripper succeeds 28/38 (baseline 4/25).
- Off-policy ablations: adding gripper status and height lifts success from 53% to 70%; clipped double Q-learning lifts it from 63% to 81%.
Limitations
Only grasping is shown. The real dataset grew over four months of separate experiments, so the controlled ablations are mostly in simulation. In dense clutter the policy sometimes regrasps repeatedly, and small objects stuck in bin corners caused many failures. The image-subtraction reward can miss small objects: 3 of 28 analyzed failures were actually successful grasps.
Videos
Figures
Abstract
In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of grasping, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execute the desired grasp, our method enables closed-loop vision-based control, whereby the robot continuously updates its grasp strategy based on the most recent observations to optimize long-horizon grasp success. To that end, we introduce QT-Opt, a scalable self-supervised vision-based reinforcement learning framework that can leverage over 580k real-world grasp attempts to train a deep neural network Q-function with over 1.2M parameters to perform closed-loop, real-world grasping that generalizes to 96% grasp success on unseen objects. Aside from attaining a very high success rate, our method exhibits behaviors that are quite distinct from more standard grasping systems: using only RGB vision-based perception from an over-the-shoulder camera, our method automatically learns regrasping strategies, probes objects to find the most effective grasps, learns to reposition objects and perform other non-prehensile pre-grasp manipulations, and responds dynamically to disturbances and perturbations.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.