AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains

Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, Cewu Lu

Seminal December 2022

Bin picking: the robot clears a plate of unseen daily objects into a bin.

TL;DR AnyGrasp predicts dense 7-DoF parallel-jaw grasps from one depth view in 100 ms, trained on real scans of only 144 objects, and tracks those grasps across frames. It clears bins of over 300 unseen objects at 93.3% success (humans with a jaw, 93.9%), at over 900 picks per hour, and catches swimming robot fish 75.5% of the time.

What they do

AnyGrasp is a grasp perception system for a parallel gripper. From a single partial point cloud it predicts dense 7-DoF grasps (rotation, center, width), and it keeps those grasps consistent across frames so the robot can catch moving objects. It trains only on real depth data: GraspNet-1Billion plus 168 new scenes, giving 268 scenes of 144 objects with analytic grasp labels.

How it works

Geometry processing. Built on GSNet. A 3D convolutional backbone finds graspable points and samples 1024 seeds. For each seed it picks an approach view, then scores 12 in-plane rotations x 5 approach depths and predicts their widths. Grasps with no room for the gripper are scored zero.
Center-of-mass awareness. An extra head predicts a stable score: the normalized distance from the gripper plane to the object’s center of gravity, assuming uniform density. Grasp scores are multiplied by (1 - stable score).
Temporal association. Each grasp gets a 256-d feature built from geometry, color and pose. Cosine similarity between features in two frames gives a correspondence matrix. It is trained with a supervised contrastive loss, using adjacent camera viewpoints of static scenes as a stand-in for motion.

Results

Bin picking over 300 unseen objects (UR5): 93.3% attempt success, against 72.2% for DexNet 4.0 and 93.9% for humans using an open-loop jaw. Grasps come out in 100 ms, and the system exceeds 900 mean picks per hour.
In real bin picking, a model trained on simulated depth with Gaussian noise does much worse than one trained on real data. It also stops finding grasps once only a few objects are left.
Fish catching (Flexiv arm, wrist L515 camera, 5 trials of 8 fish): 75.5% success, against 62.5% for nearest-grasp tracking. The system runs at 7 Hz.
On a long, heavy object, the stable score cut in-hand slips from 16 to 11 in 25 attempts.

Limitations

Accuracy drops on adversarial 3D-printed objects because the system repeats a failed grasp without using feedback. Nearly half of the fish-catching failures are slips in water. Closed-loop grasping breaks once the gripper hides the target, and there is no tactile sensing. Transparent and black objects were left out of testing. Only parallel grippers are covered.

Videos

Catching swimming robot fish with a wrist camera, at 3x speed.

Picking up thin fragments of a broken clay pot.

Figures

From Figure 4: Robot and human bin-picking setups. Objects move from the plate (blue line) to the target bin (yellow ... — From Figure 4: Robot and human bin-picking setups. Objects move from the plate (blue line) to the target bin (yellow line). The robot uses a global camera; the human-held jaw has the same fingertip material and opening width as the gripper and is used open-loop.

From Figure 2: AnyGrasp architecture. The geometry processing module predicts dense grasp poses from a partial point ... — From Figure 2: AnyGrasp architecture. The geometry processing module predicts dense grasp poses from a partial point cloud. The temporal association module gives each grasp a feature vector, and cosine similarity between features across two frames forms a correspondence matrix.

From Figure 14: Real-robot bin-picking success rate per object category when training on real data versus simulated d... — From Figure 14: Real-robot bin-picking success rate per object category when training on real data versus simulated depth with added Gaussian noise.

Abstract

As the basis for prehensile manipulation, it is vital to enable robots to grasp as robustly as humans. Our innate grasping system is prompt, accurate, flexible, and continuous across spatial and temporal domains. Few existing methods cover all these properties for robot grasping. In this paper, we propose AnyGrasp for grasp perception to enable robots these abilities using a parallel gripper. Specifically, we develop a dense supervision strategy with real perception and analytic labels in the spatial-temporal domain. Additional awareness of objects' center-of-mass is incorporated into the learning process to help improve grasping stability. Utilization of grasp correspondence across observations enables dynamic grasp tracking. Our model can efficiently generate accurate, 7-DoF, dense, and temporally-smooth grasp poses and works robustly against large depth-sensing noise. Using AnyGrasp, we achieve a 93.3% success rate when clearing bins with over 300 unseen objects, which is on par with human subjects under controlled conditions. Over 900 mean-picks-per-hour is reported on a single-arm system. For dynamic grasping, we demonstrate catching swimming robot fish in the water. Our project page is at https://graspnet.net/anygrasp.html

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong