ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio Torralba, Florian Shkurti, Liam Paull

Seminal September 2023

Mapping a Replica room: input frames, 2D segments and the fused 3D object point cloud.

TL;DR Build an open-vocabulary 3D scene graph by fusing SAM masks and CLIP features across RGB-D views, captioning each object with LLaVA and GPT-4 and letting GPT-4 label the edges, with no training. Human raters judge about 70% of node captions and 90% of edges correct, and GPT-4 planning over the graph answers negation queries far better than CLIP (R@1 0.80 vs 0.26 on Replica).

What they do

Dense maps with a vision-language feature per point scale poorly and hold no relations between objects. ConceptGraphs builds an object-centric 3D scene graph from posed RGB-D frames using only off-the-shelf 2D foundation models. Nodes are objects with a point cloud, a CLIP feature and a caption; edges are spatial relationships. The graph is written out as JSON so an LLM can plan over it.

How it works

Mapping. SAM gives class-agnostic masks; each gets a CLIP feature and is projected to 3D and denoised with DBSCAN. New detections are greedily matched to map objects by the sum of a point-overlap score and a feature-similarity score, then fused by averaging features and merging point clouds.
Captioning. LLaVA captions crops from each object’s 10 best views; GPT-4 summarizes them into one caption or marks the object invalid.
Edges. 3D bounding-box IoU gives a dense graph, pruned to a minimum spanning tree; GPT-4 labels each edge (e.g. “a on b”) from captions and positions.
Planning. GPT-4 reads the object list, returns the object most relevant to a query, and its 3D pose goes to navigation or grasping.

Results

Replica scene graphs, rated by Mechanical Turk workers: node captions are correct about 70% of the time (most errors come from LLaVA), edges about 90%, with 0-5 duplicate objects per scene.
Zero-shot 3D semantic segmentation on Replica: 40.63 mAcc and 35.95 F-mIoU, against 31.53 and 38.70 for ConceptFusion+SAM.
Text-query retrieval: LLM retrieval beats CLIP on complex queries, e.g. Replica negation R@1 0.80 vs 0.26; in a lab scan (10 queries each) affordance 1.00 vs 0.40 and negation 1.00 vs 0.00.
Robots: a Jackal navigates to objects named by abstract or visual queries, searches likely containers when objects have moved, and pushes objects an LLM deems traversable; a Spot Arm picks and places a duck plush for “cuddly quacker”.

Limitations

LLaVA captions are error-prone (many small objects become toothbrushes or scissors), and the graph misses small or thin objects and has duplicates, which hurts planning when that object matters. Building and querying the graph needs many LVLM calls and proprietary LLM inference, which can be costly.

Videos

A Jackal looks for sneakers for a Ronald McDonald outfit, misses them and searches another location.

Spot picks up the duck plush for the query "cuddly quacker".

Figures

From Figure 1: ConceptGraphs fuses class-agnostic instance masks into 3D objects, tags each with a large vision-langu... — From Figure 1: ConceptGraphs fuses class-agnostic instance masks into 3D objects, tags each with a large vision-language model, and links them into a graph of spatial relationships using a large language model. In text form the graph lets an LLM answer complex scene queries and give robots facts such as traversability, shown on wheeled and legged robots.

From Figure 2: Regions segmented from posed RGB-D images get semantic feature vectors and are projected to 3D, then a... — From Figure 2: Regions segmented from posed RGB-D images get semantic feature vectors and are projected to 3D, then associated and fused across views into a set of 3D objects. Large vision and language models caption each object and infer inter-object relations, which become the graph edges; the graph is translated to text for LLM-based task planning.

From Figure 3: A Jackal answers user queries in a lab. An LLM picks the most relevant object, an LVLM checks whether ... — From Figure 3: A Jackal answers user queries in a lab. An LLM picks the most relevant object, an LVLM checks whether it is at the expected location, and if not the LLM proposes a likely container. (Blue) For "something to wear for a space party" the robot looks for a grey NASA shirt and is sent to the laundry bag. (Red) For "footwear for a Ronald McDonald outfit" it looks for red and white sneakers and is sent to a shoe rack.

Abstract

For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer video: https://youtu.be/mRhNkQwRYnc )

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong