← Paper Digest

Infer3D: Inverse Generative Modeling Enables Robust 3D Reconstruction

Amani Kiruga, Mark Hamilton, Joshua B. Tenenbaum, William T. Freeman, Vincent Sitzmann, Yilun Du, Ayush Tewari

November 2026

TL;DR Instead of trusting a feedforward network on unfamiliar images, Infer3D searches a generative model's latent space and the camera parameters until a frozen 3D Gaussian lifter's rendering matches the input. On out-of-distribution viewpoints of ShapeNet Cars it reaches 20.29 dB PSNR, against 14.30 and 15.23 dB for two inversion baselines.

What they do

Feedforward single-image 3D reconstruction networks break when the input falls outside their training distribution, for example an unusual camera angle, a fisheye lens or real-world reflections. Infer3D treats reconstruction as analysis-by-synthesis: it searches for the scene and camera parameters that best explain the image, while keeping every hypothesis on the manifold its networks were trained on.

How it works

  • Prior. A 2D generative model (StyleGAN2 or a diffusion autoencoder) produces an in-distribution image, which a frozen lifter (Splatter Image or CATSplat) turns into 3D Gaussians.
  • Search. The latent code and rendering parameters are optimized so the rendered Gaussians match the input. A population of 600 hypotheses is pruned down to 5 over 783 iterations.
  • Per shift. Camera pose for viewpoint shifts; intrinsics plus a neural distortion field for fisheye lenses; a DINOv2 feature loss with a depth term for synthetic-to-real appearance.
  • Adaptive routing. The initial reconstruction error flags OOD inputs; only those get the full optimization.

Results

  • CO3D Hydrants under OOD poses: 18.46 dB PSNR and Chamfer 0.461, against 15.73 dB and 0.733 for Splatter Image.
  • RealCars (prior trained only on synthetic cars): mean Chamfer 0.131, against 0.219 for Splatter Image, 0.152 for Splatter Image finetuned with DINO features, 0.491 for BRFI and 0.387 for 3D GAN Inversion.
  • OOD detection reaches AUROC 0.972 (Hydrants), 0.961 (Vases) and 0.997 (RE10K) in about 286 ms; with 10% OOD inputs the routed system averages 19 s per sample.

Limitations

Run on every input, the optimizer is 0.93-2.66 dB worse than the feedforward lifter on in-distribution data. Sharpness is bounded by the lightweight generators. On CO3D the optimization uses a ground-truth depth loss to fix scale. A full OOD solve takes about 2.8 minutes, and the wrist-camera manipulation example is qualitative only.

Figures

From Figure 1: Out-of-distribution inputs with unseen camera viewpoints (top), unseen sensors (middle), and real-worl...
From Figure 1: Out-of-distribution inputs with unseen camera viewpoints (top), unseen sensors (middle), and real-world texture and lighting (bottom). Infer3D infers the underlying scene and rendering parameters despite these shifts.
From Figure 3: The test-time optimization loop. The image generator and 2D-to-3D lifter stay frozen; only the latent ...
From Figure 3: The test-time optimization loop. The image generator and 2D-to-3D lifter stay frozen; only the latent code and rendering parameters are optimized to minimize the loss between the rendered reconstruction and the OOD input.
From Figure 4: Viewpoint shift. Columns show the OOD input, the Splatter Image baseline, Infer3D and ground truth.
From Figure 4: Viewpoint shift. Columns show the OOD input, the Splatter Image baseline, Infer3D and ground truth.
From Figure 6: In-the-wild photos. Each row shows the original photo, the mask fed to the model, the Splatter Image f...
From Figure 6: In-the-wild photos. Each row shows the original photo, the mask fed to the model, the Splatter Image feedforward prediction and the Infer3D reconstruction.
Abstract

Humans can understand the 3D structure of scenes even from highly unfamiliar or extreme viewpoints. Cognitive science suggests this generalization stems from a top-down generative inference strategy. In contrast, state-of-the-art 3D reconstruction neural networks typically rely on amortized, feedforward inference. While fast, this pattern-matching approach degrades when faced with out-of-distribution (OOD) scene conditions unseen during training. Inspired by the generative nature of human vision, we propose Infer3D, an inverse generative modeling framework that recovers the latent variables of a generative model to best match the monocular input. To prevent degenerate flat outputs, we couple the generative image prior with a frozen 3D lifting network that constrains the optimization space to a valid, multi-view-consistent 3D manifold. Across a variety of controlled synthetic and real-world tasks, Infer3D demonstrates superior OOD reconstruction accuracy compared to feedforward networks trained on equal or larger datasets. Furthermore, mirroring how human perception requires more time to process confusing stimuli, we introduce an adaptive inference strategy that expends more test-time compute on ambiguous OOD examples while efficiently routing in-distribution inputs. Project website: https://amanikiruga.github.io/infer3d-site/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.