VGGT: Visual Geometry Grounded Transformer

Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, David Novotny

Seminal March 2025

32 views of a pyramid turned into a point cloud in 0.51 s.

TL;DR VGGT is a 1.2B-parameter transformer that takes one to hundreds of images of a scene and predicts cameras, depth maps, point maps and point tracks in a single forward pass. Feed-forward in about 0.2 s, it beats optimization-based methods on camera pose, e.g. 85.3 vs 78.9 AUC@30 for VGGSfM v2 (about 10 s) on unseen RealEstate10K.

What they do

Classical 3D reconstruction relies on iterative optimization such as bundle adjustment (BA), and learned methods like DUSt3R and MASt3R process only two images at a time, fusing pairs in post-processing. VGGT is one feed-forward transformer that takes one, a few or hundreds of images of a scene and predicts, for every image, camera intrinsics and extrinsics, a depth map, a point map in the first camera’s frame, and features for point tracking.

How it works

Backbone. DINOv2 patchifies each image; each frame gets a camera token and four register tokens, with separate learned tokens marking the first frame as the reference. 24 blocks alternate frame-wise and global self-attention, with no cross-attention; about 1.2B parameters.
Heads. A camera head regresses rotation quaternion, translation and field of view. A DPT head outputs depth and point maps with uncertainty, plus features for a CoTracker2-style tracker.
Training. One multi-task loss (camera, depth, point map, tracking) on 16 public 3D datasets plus synthetic artist-made assets, 2-24 frames per scene, 64 A100s for nine days.
At inference, unprojecting predicted depth with predicted cameras gives better points than the point-map head.

Results

Camera pose (10 frames, AUC@30): 85.3 on RealEstate10K (not trained on) and 88.2 on CO3Dv2 in about 0.2 s, vs 78.9 / 83.4 for VGGSfM v2 at about 10 s. Adding BA gives 93.5 / 91.8 in about 1.8 s.
DTU depth without ground-truth cameras: Overall 0.382 vs 1.741 for DUSt3R, close to MASt3R’s 0.374 with ground-truth cameras. ETH3D point maps: 0.677 vs 0.826 for MASt3R.
Not trained for two-view matching, it still beats RoMa on ScanNet-1500 (33.9 vs 31.8 AUC@5). As CoTracker’s backbone it lifts TAP-Vid RGB-S δ_avg^vis from 78.9 to 84.0.
Alternating attention beats global-only and cross-attention (ETH3D Overall 0.709 vs 0.827 and 1.061). On an H100, 100 frames take 3.12 s; 200 frames take 8.75 s and 40.6 GB.

Limitations

No support for fisheye or panoramic images. Accuracy drops under extreme input rotations, and it fails on scenes with substantial non-rigid deformation. Memory grows quickly with the number of frames, and BA post-processing still improves the feed-forward result.

Videos

Reconstructing a courtroom from 128 views.

Two non-overlapping views of a courtyard reconstructed in 0.06 s.

Figures

From Figure 1: VGGT takes up to hundreds of images and predicts cameras, point maps, depth maps and point tracks for ... — From Figure 1: VGGT takes up to hundreds of images and predicts cameras, point maps, depth maps and point tracks for all of them at once in under a second.

From Figure 2: DINO turns each image into tokens and a camera token is appended; the transformer alternates frame-wis... — From Figure 2: DINO turns each image into tokens and a camera token is appended; the transformer alternates frame-wise and global self-attention. A camera head predicts intrinsics and extrinsics, and a DPT head predicts the dense outputs.

From Figure 3: VGGT vs DUSt3R on in-the-wild images. On a single oil painting DUSt3R predicts a slightly distorted pl... — From Figure 3: VGGT vs DUSt3R on in-the-wild images. On a single oil painting DUSt3R predicts a slightly distorted plane; on two views with no overlap it fails; VGGT also handles 32 views of a repetitive desert texture.

Abstract

We present VGGT, a feed-forward neural network that directly infers all key 3D attributes of a scene, including camera parameters, point maps, depth maps, and 3D point tracks, from one, a few, or hundreds of its views. This approach is a step forward in 3D computer vision, where models have typically been constrained to and specialized for single tasks. It is also simple and efficient, reconstructing images in under one second, and still outperforming alternatives that require post-processing with visual geometry optimization techniques. The network achieves state-of-the-art results in multiple 3D tasks, including camera parameter estimation, multi-view depth estimation, dense point cloud reconstruction, and 3D point tracking. We also show that using pretrained VGGT as a feature backbone significantly enhances downstream tasks, such as non-rigid point tracking and feed-forward novel view synthesis. Code and models are publicly available at https://github.com/facebookresearch/vggt.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong