Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation

Sudharshan Suresh, Haozhi Qi, Tingfan Wu, Taosha Fan, Luis Pineda, Mike Lambeta, Jitendra Malik, Mrinal Kalakrishnan, Roberto Calandra, Michael Kaess, Joseph Ortiz, Mustafa Mukadam

Seminal December 2023

Real-world run on a bell pepper, from camera and fingertip images to the shape built and tracked online.

TL;DR NeuralFeels fuses a camera, DIGIT touch sensors on four fingertips and joint angles to learn an object's neural SDF online and track its pose while an Allegro hand rotates it. Across 70 simulated and real trials it reaches an 81% reconstruction F-score with 4.7 mm average pose drift (2.3 mm when the CAD model is given).

What they do

During in-hand manipulation the fingers hide the object from the camera, yet a robot still needs its pose and shape. NeuralFeels combines an RGB-D camera, DIGIT touch sensors on the four fingertips of an Allegro hand, and joint angles to reconstruct a novel object as a neural SDF while tracking its pose, as a proprioception-driven RL policy rotates it. They release FeelSight, 70 rotation trials with ground-truth meshes.

How it works

Frontend. SAM segments the object in the camera image, prompted by the grasp centre and by fingertip pixels from forward kinematics. A tactile transformer (21.7M parameters), trained only in TACTO simulation, turns each DIGIT image into a contact depth patch.
Touch as a small camera. Tactile and visual depth supervise the same instant-NGP SDF; touch adds only surface samples, vision adds surface and free-space samples.
Backend. Shape steps update the SDF weights over a keyframe bank. Pose steps solve a sliding-window SE(3) pose graph in Theseus with SDF, ICP and regularizer terms. Sensor poses come from kinematics and calibration, so it is the object’s field that gets posed.

Results

Novel objects: average final F-score 81% and pose drift 4.7 mm. Touch improves reconstruction by 15.3% (sim) and 14.6% (real), and pose tracking by 21.3% and 26.6%, over vision-only.
Known CAD model: average pose error around 2.3 mm; touch lowers it by 22.29% in simulation but only 3.9% on hardware.
Over 200 simulated camera views of a Rubik’s cube, touch improves tracking by 21.2% on average and by up to 94.1% at heavily occluded views.

Limitations

Each object is learned from scratch, so tracking is fragile in the first seconds, and the real-world policy may not explore the whole object. Gains from touch shrink on hardware: the real DIGIT gel gives sparser contacts and the policy sometimes needs a human to help. The camera is fixed, and results were played back at 1 Hz; the full backend runs at 5 Hz. Symmetric objects such as the peach and pear give higher pose error.

Videos

Simulated occlusion study showing how much touch improves tracking from each camera viewpoint.

The FeelSight trials, real-world and simulated in-hand rotations of different objects.

Figures

From Figure 1: NeuralFeels estimates the pose and shape of novel objects (right) during in-hand manipulation by learn... — From Figure 1: NeuralFeels estimates the pose and shape of novel objects (right) during in-hand manipulation by learning a neural field online from vision, touch and proprioception (left).

From Figure 2: The perception stack. The frontend turns camera, touch and joint-angle data into segmented visuo-tacti... — From Figure 2: The perception stack. The frontend turns camera, touch and joint-angle data into segmented visuo-tactile depth with pre-trained models; the backend trains a neural SDF on that depth while a pose graph tracks the posed field.

From Figure 3: SLAM results over 70 experiments (40 simulated, 30 real), 5 seeds each: shape and pose metrics, tracki... — From Figure 3: SLAM results over 70 experiments (40 simulated, 30 real), 5 seeds each: shape and pose metrics, tracking failures, final pose renderings, and marching-cubes reconstructions where touch completes and refines the shape.

From Figure 6: Occlusion and depth-noise ablations. (a) Tracking gain from touch for camera viewpoints on a sphere ar... — From Figure 6: Occlusion and depth-noise ablations. (a) Tracking gain from touch for camera viewpoints on a sphere around the hand, largest at occluded views. (b) Pose error against occlusion score. (c) Pose error as visual depth noise grows.

Abstract

To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily employs vision, and restricts to tracking a priori known objects. Moreover, visual occlusion of objects in-hand is imminent during manipulation, preventing current systems to push beyond tasks without occlusion. We combine vision and touch sensing on a multi-fingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We study multimodal in-hand perception in simulation and the real-world, interacting with different objects via a proprioception-driven policy. Our experiments show final reconstruction F-scores of $81$% and average pose drifts of $4.7\,\text{mm}$, further reduced to $2.3\,\text{mm}$ with known CAD models. Additionally, we observe that under heavy visual occlusion we can achieve up to $94$% improvements in tracking compared to vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step towards benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone towards advancing robot dexterity. Videos can be found on our project website https://suddhu.github.io/neural-feels/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong