Grasping
Choosing and executing grasps, from analytic grasp quality to grasping learned at scale.
Start here: seminal papers
Planning Optimal GraspsSeminal
IEEE International Conference on Robotics and Automation (ICRA), May 1992
Two grasp quality measures, one bounding the largest finger force and one bounding the total finger force, each equal to the radius of the largest origin-centred ball inside a convex hull of contact wrenches. They lead to a grasp planner for polygons held by two- and three-jaw grippers that runs in O(n) for n edges in the two-jaw case.
Robotic Grasping and Contact: A ReviewSeminal
IEEE International Conference on Robotics and Automation (ICRA), April 2000
A short survey of two decades of analytical grasping theory, covering contact models, form and force closure, grasp force distribution, stability, grasp quality measures and the dynamics of frictional contact. It names accurate, tractable models of contact compliance and friction as the key open problems.
GraspIt! A Versatile Simulator for Robotic GraspingSeminal
IEEE Robotics & Automation Magazine, December 2004
GraspIt! is a public simulator that imports hand, arm and object models, finds the contacts as a hand closes on an object, and scores the grasp on the fly with wrench-space quality measures. It ships with five hand models, a Puma arm and a mobile base, and a rigid-body dynamics engine for simulating how grasps form.
Robotic Grasping of Novel Objects using VisionSeminal
The International Journal of Robotics Research, February 2008
A classifier trained only on synthetic images of five object classes predicts where in an image to grasp an object, and the predictions from two or more views are triangulated into a 3-d grasp point without building a 3-d model. A 5-DoF arm picked up novel objects 87.8% of the time and unloaded a dishwasher with 80% success.
Deep Learning for Detecting Robotic Grasps★ Must read
January 2013
Learn grasp features from RGB-D data with deep networks instead of hand-designing them, using a small network to scan candidate grasp rectangles and a larger one to re-rank the best 100. On unseen objects it detects correct grasps 75.6% of the time vs 58.3% for hand-engineered features, and executes grasps with 84% success on Baxter and 89% on PR2.
Data-Driven Grasp Synthesis - A SurveySeminal
September 2013
A survey that sorts data-driven grasp synthesis by what the robot knows about the object beforehand, with stored grasps for known objects, grasps transferred by similarity for familiar ones and heuristics on sensor data for unknown ones. Its tables classify 22, 31 and 16 methods in these three groups.
Real-Time Grasp Detection Using Convolutional Neural NetworksSeminal
December 2014
A single AlexNet-style CNN looks once at an RGB-D image and regresses a grasp rectangle directly, instead of scoring many patches with a sliding window. Its MultiGrasp variant reaches 88.0% accuracy on the Cornell Grasping Dataset at 13 frames per second, versus 73.9% at 13.5 s per image for the previous best.
Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot HoursSeminal
September 2015
A Baxter robot collected 50K self-labeled grasp attempts over 700 hours, enough to train a CNN that scores 18 grasp angles per image patch. It reaches 79.5% accuracy on a held-out set of novel objects and lifts novel objects in 66% of real trials.
Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection★ Must read
March 2016
Train a CNN on about 800,000 self-supervised grasp attempts from up to 14 robots to predict whether a gripper motion will lead to a successful grasp, then use it to servo the gripper from an uncalibrated monocular camera. When clearing a bin of novel objects it fails on 17.5% of the first 30 grasps, against 33.7% for an open-loop variant and 50.8% for a hand-engineered depth-based system.
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics★ Must read
March 2017
Train a CNN to score parallel-jaw grasps from depth images using only synthetic data, 6.7 million rendered point clouds labeled with a robust analytic grasp metric. On an ABB YuMi it plans grasps in 0.8 s with 93% success on known adversarial objects and reaches 99% precision on 40 novel household objects.
Grasp Pose Detection in Point CloudsSeminal
June 2017
Sample 6-DOF parallel-jaw grasp candidates on any visible surface of a point cloud and score each with a small CNN over multi-view projections of the gripper's closing region. A Baxter clearing dense clutter of novel objects succeeds on 93% of 288 grasp attempts.
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic GraspingSeminal
September 2017
Train a monocular-RGB grasp-success CNN on 8 million simulated grasps of random procedural shapes plus real data, with GraspGAN making the rendered images look real and a domain-adversarial loss aligning features. With 2% of the real data it matches the all-real model (68.51% vs 67.65%), and with no real labels it reaches 63.40%.
Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image MatchingSeminal
October 2017
A pick-and-place system that first grasps whatever its pixel-wise affordance maps rank highest across four suction and parallel-jaw primitives, then identifies the held object by matching it to product images, so novel objects need no retraining. It won the 2017 Amazon Robotics Challenge stowing task, stowing all 20 objects with 100% recognition accuracy.
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?Seminal
October 2017
A network predicts whether a grasp will hold from a camera image plus two GelSight fingertip images, trained on 9,269 self-collected grasps. Adding touch lifts real-robot grasp success on 12 unseen objects from 80% (vision only) to 94%.
Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement LearningSeminal
March 2018
VPG trains two fully convolutional Q-networks, one for pushing and one for grasping, by trial and error, so pushes are learned for how much they help later grasps. On challenging real-world block arrangements it reaches 83.3% grasp success versus 43.5% for a grasping-only policy, after about 5.5 hours of robot training.
Jacquard: A Large Scale Dataset for Robotic Grasp DetectionSeminal
March 2018
A synthetic grasp dataset: 54k RGB-D images of 11k ShapeNet objects, labeled with 1.1M grasps that succeeded in a physics simulator. An AlexNet trained on it succeeds on 78.43% of real-robot grasps, against 60.46% when trained on the hand-labeled Cornell dataset.
Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis ApproachSeminal
April 2018
GG-CNN, a 62k-parameter fully convolutional network, predicts a grasp quality, angle and gripper width for every pixel of a depth image in 19 ms, fast enough to re-plan the grasp in a closed loop. It grasps objects that are moved during the attempt with 83% success on adversarial objects and 88% on household objects.
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch★ Must read
May 2018
A network that sees a camera image and two GelSight fingertip images predicts whether a candidate grasp adjustment will succeed, and the robot regrasps by picking the best one. On 11 unseen objects it grasps 94% of the time, against 76% for a hand-tuned baseline.
Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-SupervisionSeminal
June 2018
TOG-Net learns where to grasp a tool and how to swing it together, trained only on simulated trial and error with procedurally generated tools. On a real Sawyer with 9 unseen objects it reaches 71.1% success at sweeping and 80.0% at hammering.
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation★ Must read
June 2018
Off-policy Q-learning on 580k real grasp attempts from 7 robots, choosing actions by optimizing the Q-function instead of training an actor. The closed-loop, RGB-only policy grasps 96% of unseen objects, against 78% for a prior self-supervised grasping method.
PointNetGPD: Detecting Grasp Configurations from Point SetsSeminal
September 2018
PointNetGPD scores parallel-jaw grasp candidates by running a PointNet classifier on the raw points inside the gripper, trained on 350k real point clouds and grasps over YCB objects. It beats the GPD baseline at grasp-quality classification (84.75% vs 79.96% on single-view clouds), and on a real UR5 it clears more of each cluttered pile (completion 95-100% vs 81.5-95%).
TossingBot: Learning to Throw Arbitrary Objects with Residual PhysicsSeminal
March 2019
TossingBot learns by trial and error to grasp arbitrary objects from a bin and throw them into boxes beyond a UR5's reach, with a network that predicts a correction on top of a ballistic release velocity. On a real robot it reaches 514 picks per hour with 84.7% throwing accuracy on seen objects (82.3% on unseen).
6-DOF GraspNet: Variational Grasp Generation for Object ManipulationSeminal
May 2019
A variational autoencoder samples diverse 6-DOF grasps from a partial object point cloud, and a learned evaluator scores them and refines them by gradient ascent. Trained only on simulated grasps, it picks up 17 real objects with 88% success versus 47% for GPD.
Antipodal Robotic Grasping using Generative Residual Convolutional Neural NetworkSeminal
September 2019
GR-ConvNet, a 1.9M-parameter residual network, turns an RGB-D image into per-pixel grasp quality, angle and width maps in about 20 ms. It scores 97.7% on Cornell and 94.6% on Jacquard, and a Baxter arm picks 95.4% of household and 93% of adversarial objects.
Grasping in the Wild: Learning 6DoF Closed-Loop Grasping from Low-Cost DemonstrationsSeminal
December 2019
People record 12 hours of grasping demonstrations in everyday places with a $600 handheld grabber that carries a wrist RGB-D camera, and these pretrain a closed-loop 6DoF grasping Q-function that scores rendered future views. After finetuning on a UR5, it grasps novel objects 92% of the time in static scenes and 88% while the objects are being moved.
Reactive Human-to-Robot Handovers of Arbitrary ObjectsSeminal
November 2020
A vision-based system lets a Franka arm take unknown objects from a person's hand by segmenting the hand, generating temporally consistent 6-DOF grasps on the remaining points and picking among them with a reactive planner. In a six-person study it took all 10 household test objects from every participant, with an 81.8% average success rate and 1.3 attempts per object.
ACRONYM: A Large-Scale Grasp Dataset Based on SimulationSeminal
November 2020
A dataset of 17.7M parallel-jaw grasps on 8872 ShapeNetSem objects from 262 categories, each labeled by closing and shaking the gripper in the FleX physics simulator. Retraining 6-DOF GraspNet and GPD on it, with unchanged models, raises their grasp performance 2-3 fold over training on a smaller dataset.
Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in ClutterSeminal
January 2021
A 3D fully convolutional network takes a TSDF of a cluttered scene and predicts grasp quality, gripper orientation and opening width at every voxel in one 10 ms forward pass. Trained only on simulated grasps, it clears 92% of objects in real-robot clutter removal without explicit collision checking.
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes★ Must read
March 2021
A single PointNet++ network treats each observed depth point as a possible grasp contact and predicts a full 6-DoF parallel-jaw grasp there, so it only has to learn 4 DoF. Trained on simulated grasps only, it picks unseen objects from cluttered scenes on a Franka with 90.2% success, versus 80.4% for the prior best pipeline.
Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand ControlSeminal
April 2021
Predict which of 13 grasp types a prosthetic-hand user intends by multiplying the probabilities of a forearm-EMG classifier and a gaze-guided camera detector. During reaching, fusion is 95.3% accurate, against 81.64% for EMG and 80.5% for vision alone.
DexYCB: A Benchmark for Capturing Hand Grasping of ObjectsSeminal
April 2021
A marker-less dataset of 10 people grasping 20 YCB objects: 582K RGB-D frames from 8 cameras, with 3D hand and object poses fit to crowdsourced keypoints. Hand-pose models trained on it transfer to HO-3D better than the reverse (1.4-1.9x versus 3.4-3.7x error increase), and it adds a benchmark for safe human-to-robot handover grasps.
HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object HandoversSeminal
May 2022
A PyBullet benchmark for human-to-robot handovers, where a simulated hand replays 1,000 real motion captures and a Panda arm has to take the object without touching the hand. The best of four existing baselines succeeds 64.58% of the time, and the baselines rank the same way they did in a real user study.
DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on SimulationSeminal
October 2022
A faster, more robust optimizer for force-closure grasps produces DexGraspNet, 1.32 million simulation-validated ShadowHand grasps on 5355 objects. Training the DDG grasp network on it instead of the earlier 6.9K-grasp dataset raises simulated success from 57.4% to 67.5%.
AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal DomainsSeminal
December 2022
AnyGrasp predicts dense 7-DoF parallel-jaw grasps from one depth view in 100 ms, trained on real scans of only 144 objects, and tracks those grasps across frames. It clears bins of over 300 unseen objects at 93.3% success (humans with a jaw, 93.9%), at over 900 picks per hour, and catches swimming robot fish 75.5% of the time.
UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned PolicySeminal
March 2023
UniDexGrasp samples diverse ShadowHand grasp poses from a tabletop point cloud, then executes one with a goal-conditioned RL policy distilled to point-cloud input. In simulation the full pipeline lifts 58-66% of thousands of objects, including unseen categories, while the best baseline policy reaches 25% even with oracle state.
Learning Human-to-Robot Handovers from Point CloudsSeminal
March 2023
A point-cloud handover policy is trained in simulation against replayed human motion, first with the hand held still and planner demonstrations, then fine-tuned while the hand moves. It reaches 75% success on the HandoverSim test scenes (69% with the human moving) and beats GA-DDPG on a real Franka (41/60 vs 21/60 handovers for one subject).
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.