3D Shape Reconstruction from Vision and Touch

Edward J. Smith, Roberto Calandra, Adriana Romero, Georgia Gkioxari, David Meger, Jitendra Malik, Michal Drozdzal

Seminal July 2020

TL;DR Reconstructs an object's 3D mesh from one RGB image plus four simulated high-resolution touch readings from a grasping robot hand, by deforming mesh patches from vision around exact patches from touch. Adding touch lowers test Chamfer distance from 1.074 to 0.991 with the hand occluding the view, and five grasps beat one (0.654 vs 0.804).

What they do

Vision gives global shape but is blocked by the hand; touch gives exact local geometry but cannot extrapolate. This paper fuses the two for 3D shape reconstruction. A single RGB image and four touch readings from an Allegro hand with DIGIT-like sensors on each fingertip go in; a mesh of the whole object comes out. To train and test it, the authors build a simulated dataset of grasps on 1,732 bottles, knives, cellphones and rifles from 3D Warehouse.

How it works

Charts. The surface is a set of small independent mesh sheets: 95 vision charts arranged into a closed sphere, plus one touch chart per fingertip.
Touch charts. A U-Net maps each 100x100 touch image to the depth of the impression on the gel, giving a local point cloud; a touch chart is then fitted to it.
Fusion. A graph network deforms all charts three times, with image features pooled onto each vertex. Neighboring vision charts share boundary vertices, and each touch chart’s center vertex is linked to all vision-chart boundaries. Successful touch charts are reset to their measured shape after every step, so vision charts fill in around them.
Data. PyBullet places the hand on the object and closes the fingers; touch images are rendered by sphere tracing plus Phong shading, and Blender renders images with and without the hand.

Results

Test Chamfer distance: occluded vision 1.074, plus touch 0.991; unoccluded vision 0.861, plus touch 0.804; touch only 3.050.
On validation, the full model (0.751) beats sphere-based fusion (0.813), no touch-chart copying (0.756) and no chart communication (0.788).
More grasps help: with vision, 0.804 at 1 grasp to 0.654 at 5; touch only, 3.05 to 1.237.
Touch also helps beyond the sensor: at the touch site, local error drops from 4.55e-5 to 1.51e-5 with unoccluded vision and stays lower out to five sensor widths.

Limitations

Charts can overlap poorly, leaving noisy rather than smooth surfaces, and they are not forced to connect; stray charts appeared in 5% of 100 inspected test objects. Training needs full 3D ground truth and testing needs the full hand pose, which the authors call somewhat unrealistic outside simulation. All data is simulated.

Figures

From Figure 1: A simulated hand grasps an object; vision gives global context, touch gives local structure, and the t... — From Figure 1: A simulated hand grasps an object; vision gives global context, touch gives local structure, and the two are fused into a predicted object surface.

From Figure 2: Touch charts are predicted from the 4 touch readings, image features are projected onto all charts, an... — From Figure 2: Touch charts are predicted from the 4 touch readings, image features are projected onto all charts, and the charts are deformed iteratively with the touch charts held fixed.

From Figure 9: Reconstructions of a bottle from 1 grasp without vision and from 1, 3 and 5 grasps with an unoccluded ... — From Figure 9: Reconstructions of a bottle from 1 grasp without vision and from 1, 3 and 5 grasps with an unoccluded image, with the error over the true surface below each.

Abstract

When a toddler is presented a new toy, their instinctual behaviour is to pick it up and inspect it with their hand and eyes in tandem, clearly searching over its surface to properly understand what they are playing with. At any instance here, touch provides high fidelity localized information while vision provides complementary global context. However, in 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. In this paper, we study this problem and present an effective chart-based approach to multi-modal shape understanding which encourages a similar fusion vision and touch information. To do so, we introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects. Our results show that (1) leveraging both vision and touch signals consistently improves single-modality baselines; (2) our approach outperforms alternative modality fusion methods and strongly benefits from the proposed chart-based structure; (3) the reconstruction quality increases with the number of grasps provided; and (4) the touch information not only enhances the reconstruction at the touch site but also extrapolates to its local neighborhood.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong