GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger

Shaoxiong Wang, Yu She, Branden Romero, Edward Adelson

Seminal June 2021

TL;DR GelSight Wedge is a compact tactile robot finger that keeps GelSight's high-resolution 3D reconstruction, with lights on only three sides so the tip can pick up small objects. When a light direction is missing, a U-Net trained on synthetic surfaces fills in the gradient and cuts the angle error from 5.7° to 2.7°.

What they do

Earlier GelSight sensors measure high-resolution 3D contact geometry but are too bulky for robot fingers, and the slimmer GelSlim is not optically designed for photometric stereo. GelSight Wedge is a compact finger that meets both the optical and the mechanical constraints. The paper compares 3D reconstruction with 3, 2 and 1 lights, shrinks the sensor to human-finger size, and uses the reconstructed point cloud for pose tracking.

How it works

Optics. Lensless LED arrays sit on three sides of a 4.5 mm acrylic block; the tip has no light so it can pick up small objects. Gray filters cut internal reflections (a filter passing 1/4 of the light cuts them to 1/16), diffusers light the boundary, and a curved gel evens out illumination. A $16 Raspberry Pi camera (120° FOV) views the gel through a mirror tilted 30°.
Color to gradient. A small MLP (5-32-32-32-2) maps each pixel’s color and position to surface gradients. It is trained on 32 presses of a 2.4 mm-radius calibration ball, and a fast Poisson solver integrates the gradients into depth.
Missing lights. When one axis has no light, a U-Net trained on 10k synthetic random surfaces infers the missing horizontal gradient from the vertical one, with no extra real data.
Pose tracking. ICP on the reconstructed point cloud tracks a cube by its corner at about 10 Hz, given the initial pose.

Results

Reconstructions of a ball array, a Lego block, a rubber tyre, a screw cap and an M2 screw; the authors report more fidelity than GelSlim.
Gradient angle error (x/y): RGB 2.25°/2.19°, RG 2.33°/2.29°. With two opposing lights (RB) the U-Net cuts the x error from 5.71° to 2.66°; with one light, from 5.60° to 2.90°.
Marker gradients are interpolated in 10 ms at 200×150 with nearest-neighbour; linear and cubic take 60 and 70 ms.

Limitations

Markers leave small bumps unless their gradients are interpolated, and gel deformation and shadows distort the shape of sharp objects. A single light still produces artifacts. Pose tracking needs the initial pose, and its error is not analysed. The tip thickness is set by the LED size.

Videos

Figures

From Figure 1: A gripper with two GelSight Wedge sensors touches a plate; the bottom sensor squeezes into the gap bet... — From Figure 1: A gripper with two GelSight Wedge sensors touches a plate; the bottom sensor squeezes into the gap between plate and table, and its tactile imprint shows the contact geometry.

From Figure 3: Sensor design: an exploded view of the components, the assembled CAD model, and side and bottom schema... — From Figure 3: Sensor design: an exploded view of the components, the assembled CAD model, and side and bottom schematics.

From Figure 6: 3D reconstruction workflow: the raw imprint is unwarped, the blank background frame is subtracted, and... — From Figure 6: 3D reconstruction workflow: the raw imprint is unwarped, the blank background frame is subtracted, and the difference image is mapped to surface gradients and integrated into a 3D shape.

From Figure 9: Reconstructions with different light configurations. With perpendicular lights (RGB, RG) the Poisson s... — From Figure 9: Reconstructions with different light configurations. With perpendicular lights (RGB, RG) the Poisson solver alone works; with RB or R, a neural network estimates the missing horizontal gradients.

Abstract

Vision-based tactile sensors have the potential to provide important contact geometry to localize the objective with visual occlusion. However, it is challenging to measure high-resolution 3D contact geometry for a compact robot finger, to simultaneously meet optical and mechanical constraints. In this work, we present the GelSight Wedge sensor, which is optimized to have a compact shape for robot fingers, while achieving high-resolution 3D reconstruction. We evaluate the 3D reconstruction under different lighting configurations, and extend the method from 3 lights to 1 or 2 lights. We demonstrate the flexibility of the design by shrinking the sensor to the size of a human finger for fine manipulation tasks. We also show the effectiveness and potential of the reconstructed 3D geometry for pose tracking in the 3D space.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong