The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
TL;DR A network predicts whether a grasp will hold from a camera image plus two GelSight fingertip images, trained on 9,269 self-collected grasps. Adding touch lifts real-robot grasp success on 12 unseen objects from 80% (vision only) to 94%.
What they do
Does touch help a robot tell whether a grasp will hold? The authors put a GelSight tactile sensor on each finger of a parallel gripper on a Sawyer arm and autonomously collected 9,269 grasp trials on 106 objects. They train end-to-end networks that predict grasp success from vision, touch or both, then compare them on held-out objects and in real grasping.
How it works
- Inputs. Two RGB frames from a front-mounted Kinect 2, one before the grasp and one when the fingers have closed. For each GelSight the network sees the difference between the during-grasp and before-grasp images.
- Model. Each image goes through an ImageNet-pretrained ResNet-50, with weights shared within a modality. The pooled features are concatenated and a two-layer fully connected network outputs the success probability.
- Data. Grasps aim at a cylinder fitted to the object from depth, with randomized position, height, angle and gripping force. Labels come from a contact classifier on GelSight images taken two seconds after lift-off, with some manual correction.
- Grasp selection. The robot closes the gripper at random poses near the object and lifts only when the predicted success exceeds 0.9.
Results
- Prediction accuracy on unseen objects: tactile + vision 77.8%, tactile only 75.6%, depth 73.2%, hand-crafted indentation features 72.7%, RGB only 68.8%, chance 61.8%.
- Real grasping on 12 new objects, 10 trials each: 94% success with tactile + vision, 80% with vision only, 43% with the data-collection heuristic. A computer mouse was picked up 60% of the time with touch versus 10% with vision only.
- The vision-only model sometimes lifted with one finger not touching the object; the visuo-tactile model never did.
Limitations
The model can only reject a grasp after the fingers close, so with random proposals the number of attempts is unbounded: most trials took 2-3 grasps, one took 45. It ignores events after lift-off such as slip.
Figures
Abstract
A successful grasp requires careful balancing of the contact forces. Deducing whether a particular grasp will be successful from indirect measurements, such as vision, is therefore quite challenging, and direct sensing of contacts through touch sensing provides an appealing avenue toward more successful and consistent robotic grasping. However, in order to fully evaluate the value of touch sensing for grasp outcome prediction, we must understand how touch sensing can influence outcome prediction accuracy when combined with other modalities. Doing so using conventional model-based techniques is exceptionally difficult. In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch. To that end, we collected more than 9,000 grasping trials using a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger, and evaluated visuo-tactile deep neural network models to directly predict grasp outcomes from either modality individually, and from both modalities together. Our experimental results indicate that incorporating tactile readings substantially improve grasping performance.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.