Deep Learning for Detecting Robotic Grasps
TL;DR Learn grasp features from RGB-D data with deep networks instead of hand-designing them, using a small network to scan candidate grasp rectangles and a larger one to re-rank the best 100. On unseen objects it detects correct grasps 75.6% of the time vs 58.3% for hand-engineered features, and executes grasps with 84% success on Baxter and 89% on PR2.
What they do
Grasp detection from a single RGB-D image: find the oriented rectangle where a parallel-plate gripper should close. Earlier detectors ranked candidate rectangles with hand-designed features. This paper learns the features with a deep network, adds a regularizer suited to mixed color and depth input, and uses a two-stage cascade to keep the search over rectangles affordable.
How it works
- Input. A grasp is a 2D rectangle (position, width, height, angle). The patch inside it is rotated to align with the gripper plates and padded into a 24x24 window with seven channels: YUV color, depth and surface normals. Padding is masked out and the remaining inputs are scaled up, which removes a bias toward square rectangles.
- Network. Two sigmoid hidden layers, pretrained as sparse auto-encoders, feed a logistic classifier that outputs the probability a rectangle is graspable.
- Multimodal regularization. Instead of concatenating modalities or learning separate features per modality, a group penalty pushes each first-layer feature to use only a few modalities.
- Cascade. A 50-unit network searches exhaustively; a 200-unit network re-ranks its top 100 rectangles. The winner is turned into a grasp point plus an approach vector from the surface normal.
Results
- Cornell grasping dataset (1035 images, 280 objects): recognition accuracy 93.7% vs 84.7% for the hand-engineered features of Jiang et al.; detection 75.6% on unseen objects vs 58.3%.
- Group regularization beats plain L1 by up to 1.8%, and the cascade beats the large network alone by up to 2.4% while running it on about 1000x fewer rectangles (24.6 s to 13.5 s per image in MATLAB).
- Real robots, 30 objects: 84% success over 100 trials on Baxter and 89% on PR2. A center-of-mass baseline succeeded 31% of the time.
Limitations
Only parallel-plate grippers, and the 3D gripper orientation comes from the surface normal rather than being searched. Features are purely local, so the detector cannot prefer semantically right regions such as handles. The dataset has only white backgrounds, which caused failures on a white-and-yellow box, and glossy objects with no Kinect depth could not be grasped.
Figures
Abstract
We consider the problem of detecting robotic grasps in an RGB-D view of a scene containing objects. In this work, we apply a deep learning approach to solve this problem, which avoids time-consuming hand-design of features. This presents two main challenges. First, we need to evaluate a huge number of candidate grasps. In order to make detection fast, as well as robust, we present a two-step cascaded structure with two deep networks, where the top detections from the first are re-evaluated by the second. The first network has fewer features, is faster to run, and can effectively prune out unlikely candidate grasps. The second, with more features, is slower but has to run only on the top few detections. Second, we need to handle multimodal inputs well, for which we present a method to apply structured regularization on the weights based on multimodal group regularization. We demonstrate that our method outperforms the previous state-of-the-art methods in robotic grasp detection, and can be used to successfully execute grasps on two different robotic platforms.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.