Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor

Wenzhen Yuan, Chenzhuo Zhu, Andrew Owens, Mandayam A. Srinivasan, Edward H. Adelson

Seminal April 2017

A human presses GelSight on a silicone sample while the sensor's camera view is shown.

TL;DR A CNN+LSTM regresses object hardness directly from GelSight tactile videos of loosely controlled presses by a human or a robot gripper, without knowing the object's shape. On trained shapes with unseen hardness values it reaches R² 0.96 (RMSE 5.18 Shore 00); on novel shapes R² drops to 0.79.

What they do

Robots usually measure hardness by pressing with strictly controlled motion on objects of known geometry. The authors’ earlier GelSight model handled only hemispheres and needed the radius as input. Here a network regresses hardness (Shore 00) directly from the GelSight video of one press, made by hand or by a robot gripper with unknown force and trajectory, on objects of many shapes.

How it works

Sensor. A fingertip GelSight with a dome-shaped, 17 Shore 00 elastomer and black markers. Its camera records the gel at 30 Hz, showing contact geometry and, through marker motion, approximate force.
Input. Five frames from the loading phase of a press. The first is where mean image intensity crosses a threshold and the last is the peak intensity change. The first frame is subtracted from all of them.
Network. Per-frame VGG16 fc7 features feed an LSTM. Per-frame hardness outputs from the last 3 frames are averaged; the loss is a Huber loss.
Data. About 7000 human presses on silicone samples from 8 to 87 Shore 00: hemispheres, cylinders, flats, edges, corners and chocolate-mold shapes. Each video is reused with earlier end points to mimic smaller maximum forces.

Results

Trained shapes, unseen hardness: R² 0.9564, RMSE 5.18 over 1398 videos.
Novel cylinder and sphere shapes: R² 0.7868, RMSE 11.05 (73 videos). A Weiss WSG 50 gripper on trained samples gives R² 0.8524, RMSE 10.28 (683 videos), although training used only human presses.
Unseen arbitrary shapes: R² 0.57 (RMSE 19.3) for simple shapes and 0.39 (RMSE 18.2) for complicated ones.
On tomatoes of different ripeness, predictions are similar to human hardness rankings. Rigid objects are estimated above 80.

Limitations

Objects with sharp curvatures, multiple ridges or textured surfaces, such as ridged tubes or a candy with sugar particles on its surface, are estimated as too hard. The authors attribute this to an incomplete training set. Because the gel is soft, the sensor cannot tell hardness apart above about 70 Shore 00. Natural objects have no ground-truth hardness, only human rankings.

Videos

Pressing a hard and a soft sample side by side; the softer sample flattens more under the press.

With a marker-covered gel, the markers move further on the hard sample, indicating larger force.

Figures

From Figure 1: The GelSight sensor and silicone test samples. Either a human tester presses the sensor on a sample (b... — From Figure 1: The GelSight sensor and silicone test samples. Either a human tester presses the sensor on a sample (b), or a robot gripper squeezes the sample with the sensor (c). The bottom-right images are GelSight views of a soft and a hard cylinder: color shows the surface normal, and the displacement of the black markers relates to contact force.

From Figure 2: GelSight pressing a soft and a hard sample of the same shape. The soft sample deforms more and its rid... — From Figure 2: GelSight pressing a soft and a hard sample of the same shape. The soft sample deforms more and its ridges flatten; the hard sample deforms less and its ridges stay sharp. Brighter color means a larger height gradient, and the arrow field of the black markers indicates contact force.

From Figure 3: The network maps a GelSight video to a hardness value. Five frames from the press, each with the initi... — From Figure 3: The network maps a GelSight video to a hardness value. Five frames from the press, each with the initial frame subtracted, are encoded with VGG16 fc7 features and fed to an LSTM; only the outputs of the last 3 frames are used to estimate hardness.

From Figure 6: Estimated vs. ground-truth hardness (Shore 00) for shapes seen in training with hardness values that w... — From Figure 6: Estimated vs. ground-truth hardness (Shore 00) for shapes seen in training with hardness values that were not.

Abstract

Hardness is among the most important attributes of an object that humans learn about through touch. However, approaches for robots to estimate hardness are limited, due to the lack of information provided by current tactile sensors. In this work, we address these limitations by introducing a novel method for hardness estimation, based on the GelSight tactile sensor, and the method does not require accurate control of contact conditions or the shape of objects. A GelSight has a soft contact interface, and provides high resolution tactile images of contact geometry, as well as contact force and slip conditions. In this paper, we try to use the sensor to measure hardness of objects with multiple shapes, under a loosely controlled contact condition. The contact is made manually or by a robot hand, while the force and trajectory are unknown and uneven. We analyze the data using a deep constitutional (and recurrent) neural network. Experiments show that the neural net model can estimate the hardness of objects with different shapes and hardness ranging from 8 to 87 in Shore 00 scale.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong