More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch

Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward H. Adelson, Sergey Levine

★ Must read May 2018

TL;DR A network that sees a camera image and two GelSight fingertip images predicts whether a candidate grasp adjustment will succeed, and the robot regrasps by picking the best one. On 11 unseen objects it grasps 94% of the time, against 76% for a hand-tuned baseline.

What they do

Most learned grasping picks a grasp before touching the object and cannot react to contact. Here a Sawyer arm with a parallel gripper carries a GelSight optical tactile sensor on each finger. A deep network predicts whether a candidate adjustment of the current grasp (move, rotate, change grip force) will lead to a successful grasp. The robot repeatedly executes the best-scoring adjustment and lifts once predicted success is high enough.

How it works

Model. The frontal RGB image and both GelSight images (difference before and after contact) each pass through an ImageNet-pretrained ResNet-50, with tied tactile weights. A 5-D action (3D motion, in-plane rotation, force change) plus the gripper pose go through an MLP, and a fusion MLP outputs success probability.
Data. Self-supervised: 6,450 random grasp trials on 65 training objects with random position, height, orientation and force. Labels come from a GelSight contact classifier, with manual checks on ambiguous cases; augmentation gives 18,070 examples.
Regrasping. Stochastic search over 5000 sampled actions (about 0.6 s). The robot lifts when the Platt-scaled success probability exceeds 0.9.
Minimum force. Same model, new objective: the lowest-force action with predicted success above 90%.

Results

Outcome prediction (3-fold cross-validation by object): 80.3% for tactile+vision, 79.3% tactile only, 73.0% vision only, 76.4% without the action, 62.8% chance.
Robot, 11 unseen “Easy” objects: 94.0% success vs 75.9% for cylinder fitting and 63.2% vision only. On 11 “Hard” objects, after retraining on 25,404 on-policy datapoints: 73.6% vs 66.8% and 50%.
Minimum-force grasps on a tea cup: 94/100 vs 95/100 successes, with mean force 10 N vs 20 N.
Almost 50% of z-actions in successful grasps used the maximum 2 cm downward motion.

Limitations

The model makes single-step predictions, takes no information-gathering actions, and uses coarse actions that cannot react to slip during lift-off. Experiments use single objects, not clutter. The first model did poorly on the “Hard” set and needed more on-policy data.

Videos

Figures

From Figure 1: A gripper with GelSight sensors on both fingers adjusting its grasp on an object, driven by raw visual... — From Figure 1: A gripper with GelSight sensors on both fingers adjusting its grasp on an object, driven by raw visual and tactile input.

From Figure 3: The two GelSight images and the RGB image each go through a ResNet, the action through an MLP; the fea... — From Figure 3: The two GelSight images and the RGB image each go through a ResNet, the action through an MLP; the features are concatenated and an MLP outputs the probability of a successful grasp after the action.

From Figure 5: Grasps where the model predicts moving down will (a) help or (b) not help, shown as zoomed RGB and cha... — From Figure 5: Grasps where the model predicts moving down will (a) help or (b) not help, shown as zoomed RGB and change in GelSight intensity. Moving down helps when the fingers hold the top of the object.

From Figure 8: Forces in successful Tactile+Vision grasps, means dashed. Maximum-success and minimum-force policies s... — From Figure 8: Forces in successful Tactile+Vision grasps, means dashed. Maximum-success and minimum-force policies succeed 95% and 94% of the time, but mean force drops from 20 N to 10 N.

Abstract

For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this paper, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model -- a deep, multimodal convolutional network -- predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors, nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6,450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at (i) estimating grasp adjustment outcomes, (ii) selecting efficient grasp adjustments for quick grasping, and (iii) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong