DUSt3R: Geometric 3D Vision Made Easy

Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, Jerome Revaud

★ Must read December 2023

TL;DR A transformer takes two images with unknown cameras and regresses a 3D point for every pixel of both, in one shared frame; a global alignment step merges many pairs into a full scene. With no task-specific training it reaches 96.2 RRA@15 on CO3Dv2 multi-view pose, against 80.5 for PoseDiffusion.

What they do

Classical reconstruction chains matching, pose estimation, triangulation and multi-view stereo, and needs camera parameters before dense reconstruction. DUSt3R replaces the chain with one network: given two images with unknown intrinsics and poses, it regresses a pointmap for each (one 3D point per pixel), both in the first camera’s frame. Depth, matches, focal length and relative pose are read off the pointmaps, and a global alignment extends this to many images.

How it works

Network. A shared ViT-Large encoder, two ViT-Base decoders that cross-attend to each other, and a DPT head per view that outputs a pointmap and a confidence map. Weights start from CroCo pretraining.
Training. A confidence-weighted 3D regression loss on scale-normalized pointmaps, over 8.5M image pairs from eight indoor, outdoor, synthetic and object-centric datasets.
Readouts. Matches are mutual nearest neighbours in 3D; focal length comes from a small optimization, relative pose from PnP-RANSAC, and monocular depth from feeding one image twice.
Global alignment. Pairs from a connectivity graph (about 40 ms each on an H100) get a pose and scale optimized so all pointmaps agree in 3D. Gradient descent converges in a few hundred steps, in seconds.

Results

Multi-view pose, 10 frames: on CO3Dv2, RRA@15 96.2 and mAA(30) 76.7, against 80.5 and 66.5 for PoseDiffusion. On RealEstate10K, unseen in training, mAA(30) 67.7 against 48.0 for PoseDiffusion trained on CO3Dv2 (about 80 when trained on RealEstate10K).
Zero-shot monocular depth on NYUv2: AbsRel 6.50, against 11.59 for zero-shot SlowTv and 5.40 for supervised DPT-BEiT.
Multi-view depth without ground-truth poses, depth range or intrinsics: average rel 4.73 over five test sets, against 9.3 for COLMAP with ground-truth poses, and best on ETH3D; 0.13 s against about 3 min.

Limitations

Regression is less precise than sub-pixel triangulation: on DTU the overall error is 1.7 mm, against 0.30-0.78 mm for methods given ground-truth cameras. Outputs are only defined up to scale. With unknown query intrinsics, median localization error on Cambridge grows to 64-245 cm, because sparse ground-truth pointmaps prevent reliable scaling. Each pixel is assumed to see a single 3D point, so translucent surfaces are not modeled.

Figures

From Figure 1: Given an image collection with unknown camera poses and intrinsics, DUSt3R outputs corresponding point... — From Figure 1: Given an image collection with unknown camera poses and intrinsics, DUSt3R outputs corresponding pointmaps, from which camera calibration, monocular and multi-view depth, pixel correspondences, camera poses and a dense 3D reconstruction can be recovered.

From Figure 2: Architecture. Two views are encoded by a shared ViT encoder, two transformer decoders exchange informa... — From Figure 2: Architecture. Two views are encoded by a shared ViT encoder, two transformer decoders exchange information via cross-attention, and two regression heads output a pointmap and a confidence map per view, both expressed in the coordinate frame of the first image.

From Figure 3: A scene never seen during training, after global alignment. Left to right: RGB, depth map, confidence ... — From Figure 3: A scene never seen during training, after global alignment. Left to right: RGB, depth map, confidence map, and the reconstruction with its recovered cameras.

Abstract

Multi-view stereo reconstruction (MVS) in the wild requires to first estimate the camera parameters e.g. intrinsic and extrinsic parameters. These are usually tedious and cumbersome to obtain, yet they are mandatory to triangulate corresponding pixels in 3D space, which is the core of all best performing MVS algorithms. In this work, we take an opposite stance and introduce DUSt3R, a radically novel paradigm for Dense and Unconstrained Stereo 3D Reconstruction of arbitrary image collections, i.e. operating without prior information about camera calibration nor viewpoint poses. We cast the pairwise reconstruction problem as a regression of pointmaps, relaxing the hard constraints of usual projective camera models. We show that this formulation smoothly unifies the monocular and binocular reconstruction cases. In the case where more than two images are provided, we further propose a simple yet effective global alignment strategy that expresses all pairwise pointmaps in a common reference frame. We base our network architecture on standard Transformer encoders and decoders, allowing us to leverage powerful pretrained models. Our formulation directly provides a 3D model of the scene as well as depth information, but interestingly, we can seamlessly recover from it, pixel matches, relative and absolute camera. Exhaustive experiments on all these tasks showcase that the proposed DUSt3R can unify various 3D vision tasks and set new SoTAs on monocular/multi-view depth estimation as well as relative pose estimation. In summary, DUSt3R makes many geometric 3D vision tasks easy.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong