Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation

William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, Phillip Isola

Seminal July 2023

Out-of-distribution language queries such as "Baymax", shown with their CLIP heatmaps, 5x speed.

TL;DR F3RM distills DINO or CLIP image features into a NeRF of the tabletop and finds 6-DOF grasps and placements on unseen objects from two demonstrations per task or from a text query. CLIP ResNet features succeed in 39/50 grasp and place trials, and language-guided grasping succeeds on 31/50 queries.

What they do

2D foundation-model features carry semantics but little 3D geometry. F3RM (Feature Fields for Robotic Manipulation) distills dense DINO or CLIP features into a NeRF of the tabletop, so every 3D point has both a density and a semantic feature. Grasp and place poses are found by matching the local feature field to two demonstrations; with CLIP features, the target object can instead be named in free text.

How it works

Scan and distill. A Franka Panda sweeps a RealSense camera on a selfie stick to take 50 RGB images (depth unused). A hash-grid NeRF and feature field train in at most 90 s on an RTX 3090. Dense CLIP features come from the MaskCLIP reparameterization, without fine-tuning CLIP.
Pose representation. 100 query points in the gripper frame sample the alpha-weighted feature field. Their concatenated features, averaged over two demos labeled in VR, form a task embedding.
Pose inference. Voxels that are free space or dissimilar to the task are pruned, 8 rotations are sampled per voxel, and Adam minimizes the feature distance to the task embedding. Colliding poses are rejected and the best pose with a feasible motion plan is executed.
Language. The text query retrieves the two closest demos, drops voxels closer to randomly sampled negative object names, and weights the pose cost by text-feature similarity.

Results

Five grasp/place tasks, ten scenes each with novel objects and distractors: CLIP ResNet 39/50, CLIP ViT 34/50, DINO ViT 31/50, against 27/50 for NeRF density alone and 15/50 for MIRA.
DINO captures part geometry but is confused by distractors that look like demo objects; 21 of 27 CLIP failures are grasp rotation or translation errors.
Language: 31/50 queries succeed over 13 scenes, including 9/10 on new categories such as bowls and tape, with demos only on mugs and screwdrivers. Relational and general queries succeed 4/10 each.

Limitations

Each scene needs a 1 min 40 s scan plus about 90 s of modeling before acting; the authors point to fast few-view NeRFs as the route to closed-loop, dynamic manipulation. CLIP behaves like a bag of words, causing 13 of 19 language failures on queries with attributes or relations such as “black screwdriver”. Query points are tuned by hand per task, and the authors call the 31/50 language success rate far from practical for industrial use.

Videos

Grasping unseen test mugs among distractor objects, 5x speed.

Grasping the caterpillar toy when deformed and when buried in a cluttered pile, 5x speed.

Figures

From Figure 1: (1) The robot scans the scene with a camera on a selfie stick. (2) Dense features from a 2D foundation... — From Figure 1: (1) The robot scans the scene with a camera on a selfie stick. (2) Dense features from a 2D foundation model are distilled into a feature field alongside a NeRF. (3) Querying the CLIP feature field with language gives heatmaps and 6-DOF grasps on novel objects, given only ten demonstrations.

From Figure 2: (a) A gripper pose is recorded in VR on an example mug. (b) A fixed set of query points in the gripper... — From Figure 2: (a) A gripper pose is recorded in VR on an example mug. (b) A fixed set of query points in the gripper frame samples the local feature field. (c) Features at the query points are concatenated and averaged over two demonstrations into a task embedding.

From Figure 3: Language-guided pose optimization. Grasp proposals minimize the pose cost against the selected demo, w... — From Figure 3: Language-guided pose optimization. Grasp proposals minimize the pose cost against the selected demo, weighted by the similarity between the CLIP text features and the features at the query points.

From Figure 7: Heatmaps for each language query (top) and the robot grasping the objects one after another without re... — From Figure 7: Heatmaps for each language query (top) and the robot grasping the objects one after another without rescanning (bottom). The "blue screwdriver" heatmap bleeds onto the blue bowl, showing CLIP behaving like a bag of words.

Abstract

Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using features distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong