Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning
TL;DR VPG trains two fully convolutional Q-networks, one for pushing and one for grasping, by trial and error, so pushes are learned for how much they help later grasps. On challenging real-world block arrangements it reaches 83.3% grasp success versus 43.5% for a grasping-only policy, after about 5.5 hours of robot training.
What they do
Grasping fails when objects are packed tightly together, and pushing goals such as “separate these two” are hard to write down as an objective. VPG (Visual Pushing for Grasping) learns pushing and grasping jointly with model-free Q-learning on a UR5 with a parallel-jaw gripper, so a push is valued by whether it leads to successful grasps.
How it works
- State and actions. The RGB-D camera view becomes a 224x224 top-down heightmap with 2 mm pixels. A pixel plus one of 16 rotations defines an action: a straight 10 cm push starting there, or a top-down grasp centered there.
- Q-networks. Two FCNs, each with DenseNet-121 towers on color and depth, output pixel-wise Q maps for pushing and grasping. The robot executes the maximum over all 32 maps, about 1.6M candidate actions.
- Rewards and training. Reward 1 for a successful grasp and 0.5 for a push that visibly changes the scene, with discount 0.5. Training is self-supervised: blocks are dropped on the table and the robot works until it is empty.
Results
- Simulation, 11 challenging arrangements: completion 82.7% versus 40.6% for grasping-only and 48.2% for a reactive push-and-grasp baseline; grasp success 77.2% versus 51.7% and 59.0%.
- Real robot, 7 challenging arrangements: completion 71.4% versus 42.9%, grasp success 83.3% versus 43.5%, action efficiency 69.0% versus 43.5% for grasping-only.
- Real-world training takes under 2000 transitions, about 5.5 h of robot time.
- With no push reward at all, grasp success still reaches 70-80% in simulation; removing the depth input lowers final grasp performance by about 15%.
Limitations
Primitives are parameterized on the heightmap grid, which rules out more expressive motions such as dynamic pushes or pushing and grasping at once. Training uses only toy blocks: the policy handles novel objects of similar shape but struggles with new shapes and reflective objects that give no depth. Only pushing and grasping are studied.
Videos
Figures
Abstract
Skilled robotic manipulation benefits from complex synergies between non-prehensile (e.g. pushing) and prehensile (e.g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free. In this work, we demonstrate that it is possible to discover and learn these synergies from scratch through model-free deep reinforcement learning. Our method involves training two fully convolutional networks that map from visual observations to actions: one infers the utility of pushes for a dense pixel-wise sampling of end effector orientations and locations, while the other does the same for grasping. Both networks are trained jointly in a Q-learning framework and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. In this way, our policy learns pushing motions that enable future grasps, while learning grasps that can leverage past pushes. During picking experiments in both simulation and real-world scenarios, we find that our system quickly learns complex behaviors amid challenging cases of clutter, and achieves better grasping success rates and picking efficiencies than baseline alternatives after only a few hours of training. We further demonstrate that our method is capable of generalizing to novel objects. Qualitative results (videos), code, pre-trained models, and simulation environments are available at http://vpg.cs.princeton.edu
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.