UGG: Unified Generative Geometry Model

Amani Kiruga, Zhiyi Li, Ruojin Cai, Hansen Lillemark, Xiaoshan Wu, Ayush Tewari, Yilun Du

November 2026

Depth and camera pose predicted from an in-the-wild video.

TL;DR One video diffusion transformer models RGB, depth and camera rays jointly, so camera-controlled video generation, depth estimation and pose estimation become the same fill-in-the-missing-modality task. It cuts dense camera trajectory error on ScanNet++ from 0.190 (Geo4D) to 0.087, and sampling 16 candidates at test time improves it further.

What they do

Feedforward models predict depth and camera pose from video; generative models synthesize new views. UGG treats both as queries to one generative model of a 4D scene. RGB video, depth and camera pose are three aligned modalities, and whichever ones are observed serve as context while the model denoises the rest. The same weights do camera-controlled video generation, depth estimation and pose estimation.

How it works

Aligned modalities. Camera pose is encoded as a dense map of Plücker rays, so it is image-like and lines up with RGB and depth tokens. A mixture-of-transformers backbone, initialized from Wan2.2-TI2V-5B, shares self-attention across modalities and keeps separate projections and feed-forward layers for each.
Noise as masking. Diffusion Forcing is extended across modalities and time: observed tokens get zero noise, missing ones are denoised. Losses are masked for missing annotations, so RGB-only, RGB-pose and RGB-depth datasets train one model.
Test-time search. The model samples 16 scene completions and keeps the one that best agrees with RAFT correspondences between the observed frames. No ground-truth geometry is used.

Results

Dense pose estimation: lowest ATE and translation RPE of all baselines on all five datasets, e.g. ATE 0.087 against 0.190 for Geo4D on ScanNet++. Sparse pose (first and last frame only): lowest errors on all five.
Depth: dense Abs Rel of 0.073 on ScanNet++ against 0.106 for Geo4D; in the sparse setting it beats the baselines on 4 of 5 datasets. Geo4D stays ahead on dense Sintel and 7-Scenes.
Camera-controlled generation from one view: 21.79 dB PSNR on RealEstate10K against 19.58 for SEVA. SEVA is slightly better on two-view DL3DV.
Test-time search improves sparse depth and sparse pose on all five benchmarks. Removing RGB generation from training raises KITTI depth Abs Rel from 0.152 to 0.322.

Limitations

Not uniformly better than specialized methods: task-specific view-synthesis models win on pixel-aligned metrics and specialized geometry models on several indoor depth metrics. The search criterion, RGB reprojection error, is only a proxy for accuracy and is not uniformly reliable. Training ran at 288x384 on 50-frame clips, and the experiments are on vision benchmarks, not robots.

Videos

Image to 4D and camera trajectory to 4D: novel views, depth and a 3D reconstruction from a single image.

Figures

From Figure 1: Visual generation and geometric prediction as conditional completion in a single joint model over RGB,... — From Figure 1: Visual generation and geometric prediction as conditional completion in a single joint model over RGB, depth, and camera pose. Given arbitrary partial observations, ranging from a text prompt to RGB sequences with geometric cues, the model infers a complete 4D scene representation.

From Figure 2: Generative-predictive transformer. RGB, depth, and camera pose are spatially aligned tokens, with came... — From Figure 2: Generative-predictive transformer. RGB, depth, and camera pose are spatially aligned tokens, with camera pose represented as dense ray maps. Partial observations are zero-noise context tokens, while missing modalities are noisy tokens to be denoised. A unified transformer processes all tokens with modality-specific projections and feed-forward layers together with shared multimodal self-attention.

From Figure 3: Conditional completion with noise as masking. Each task is specified by the clean/noisy pattern over R... — From Figure 3: Conditional completion with noise as masking. Each task is specified by the clean/noisy pattern over RGB, depth, camera-ray, and text tokens. Clean tokens provide observed context, while noisy tokens are denoised as missing targets.

From Figure 7: Test-time search with increasing sample budget. Up to 16 candidate completions are sampled and one is ... — From Figure 7: Test-time search with increasing sample budget. Up to 16 candidate completions are sampled and one is selected using the image-based depth or pose verifier. The dashed line shows the single-sample result, and the solid curve shows the selected result as the number of candidates increases.

Abstract

Understanding 4D scenes is a core challenge in visual intelligence, requiring models to jointly capture scene geometry, appearance, and dynamics from partial observations. However, existing approaches are subdivided into feedforward models for direct geometric prediction, and generative models for visual synthesis. We introduce UGG, a unified framework for 4D scene understanding that jointly models RGB video and geometric representations such as depth and camera pose. By assigning different noise levels across modalities and time, UGG formulates visual generation and geometric prediction as conditional completion within a single generative model. This flexible formulation enables the model to learn from diverse, heterogeneous datasets with different subsets of geometric and visual annotations. Furthermore, our generative formulation enables inference-time search over multiple plausible scene hypotheses, using geometric evidence extracted from observed RGB frames to improve geometric prediction without ground truth geometry. We demonstrate UGG's effectiveness over non-unified baselines across camera-controlled video generation, depth estimation, and camera pose prediction, including in challenging sparse observation settings. Code and visualizations are available at https://ugg-model.github.io/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong