LERF: Language Embedded Radiance Fields

Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, Matthew Tancik

Seminal March 2023

Teaser: typed queries such as "lego bulldozer" and "nerf gun" light up the matching objects in the 3D scene.

TL;DR LERF distills multi-scale CLIP embeddings into a NeRF, so a scene captured with a phone can be queried with free-form text and returns 3D relevancy maps in real time. It localizes 80.3% of 72 long-tail objects across 5 scenes, against 54.8% for OWL-ViT and 18.0% for LSeg distilled into 3D.

What they do

A NeRF captures how a scene looks but not what is in it. LERF trains a language field alongside the NeRF, supervised by CLIP embeddings of image crops. A user can then type any query, from “yellow” and “electricity” to “jake from adventure time”, and get a 3D relevancy map. There are no region proposals or masks, and CLIP is not fine-tuned on a segmentation dataset.

How it works

A field over volumes. CLIP embeds whole crops, not pixels, so the field takes a position plus a physical scale (the side of a cube) and outputs a 512-d CLIP vector. Along a ray the scale grows with distance; embeddings are averaged with the NeRF rendering weights and normalized.
Multi-scale supervision. For every training view a pyramid of CLIP crop embeddings (7 scales, 50% overlap) is precomputed. Each rendered ray is pulled toward the interpolated crop embedding by a cosine-similarity loss.
DINO regularization. A DINO head that shares the language hashgrid smooths patchy relevancy maps. Language and DINO gradients do not change the NeRF’s color or density.
Querying. Relevancy compares the rendered embedding with the query against canonical phrases (“object”, “things”, “stuff”, “texture”). Each query uses the best of 30 scales between 0 and 2 m. Training takes 45 minutes (30k steps) on an A100 in Nerfstudio.

Results

Tested on 13 hand-held iPhone scenes, including a kitchen, a grocery store and a bookstore.
Localization of 72 objects across 5 scenes: LERF 80.3%, OWL-ViT on rendered views 54.8%, LSeg distilled into 3D 18.0%. OWL-ViT does better on the ramen scene (92.5% vs 62.5%).
Existence detection: LSeg in 3D is close to LERF on COCO labels but falls well behind on long-tail labels.
Removing DINO, or training at a single 15% crop scale, visibly degrades the relevancy maps.

Limitations

LERF inherits CLIP’s bag-of-words behavior (“not red” is similar to “red”), weak spatial reasoning and false positives on look-alikes. It needs calibrated, NeRF-quality captures. Floaters and reflective or transparent objects corrupt the embeddings, and objects flush against surfaces blur into their surroundings. Each query uses a single scale, which fails for queries like “table” that need several.

Videos

Kitchen scene: RGB render beside the relevancy map for queries such as "vegetable oil".

Figurines scene: relevancy for long-tail queries such as "tesla door handle".

Figures

From Figure 1: LERF grounds CLIP in a dense, multi-scale 3D field reconstructed from a hand-held phone capture. Text ... — From Figure 1: LERF grounds CLIP in a dense, multi-scale 3D field reconstructed from a hand-held phone capture. Text queries render relevancy maps for abstract concepts ("Electricity"), visual properties ("Yellow"), long-tail objects ("Waldo") and text on a mug ("Boops").

From Figure 2: Left, the language field takes a position and a scale; CLIP embeddings are averaged along each ray wit... — From Figure 2: Left, the language field takes a position and a scale; CLIP embeddings are averaged along each ray with the NeRF rendering weights, and the scale grows with distance from the camera. Right, a multi-scale pyramid of CLIP embeddings of image crops is precomputed over the training views and interpolated to supervise each ray.

From Figure 6: Localizing long-tail objects ("cookbooks", "olive oil") from novel views with LSeg in 3D (red), OWL-ViT (green) and LERF (blue).

From Figure 4: Relevancy from patchwise 2D CLIP embeddings (left) and rendered from LERF (right). Fusing many views a... — From Figure 4: Relevancy from patchwise 2D CLIP embeddings (left) and rendered from LERF (right). Fusing many views aligns the 3D relevancy maps better with the scene geometry.

Abstract

Humans describe the physical world using natural language to refer to specific 3D locations based on a vast range of properties: visual appearance, semantics, abstract associations, or actionable affordances. In this work we propose Language Embedded Radiance Fields (LERFs), a method for grounding language embeddings from off-the-shelf models like CLIP into NeRF, which enable these types of open-ended language queries in 3D. LERF learns a dense, multi-scale language field inside NeRF by volume rendering CLIP embeddings along training rays, supervising these embeddings across training views to provide multi-view consistency and smooth the underlying language field. After optimization, LERF can extract 3D relevancy maps for a broad range of language prompts interactively in real-time, which has potential use cases in robotics, understanding vision-language models, and interacting with 3D scenes. LERF enables pixel-aligned, zero-shot queries on the distilled 3D CLIP embeddings without relying on region proposals or masks, supporting long-tail open-vocabulary queries hierarchically across the volume. The project website can be found at https://lerf.io .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong