Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Adapting Monocular Depth Estimation in Novel Scenes through Disparity Refinement and Robot Ego-Motion

Dimitrios Chatziparaschis, Georgios Georgakis, Adnan I. Ansar, Shehryar Khattak, Konstantinos Karydis

November 2026

TL;DR EM-ViT turns a frozen Depth Anything V2 relative-depth map into metric depth by training, online on a Jetson Orin, a small head that refines disparity and predicts one "virtual stereo baseline", supervised by sparse LiDAR and robot odometry. Across garage, citrus-farm and KITTI sequences it cuts mean depth error within 10 m by 32% versus online fine-tuning of metric DAV2, with 16% less training time.

What they do

Depth foundation models give good relative depth but no reliable metric scale in new scenes. EM-ViT adapts one while the robot moves: Depth Anything V2 (ViT-S) stays frozen, and a 912K-parameter head is retrained on a Jetson AGX Orin every 10 s from a few image-LiDAR pairs and odometry collected along the way. It is tested on garage data from a Boston Dynamics Spot, Jackal sequences from the CitrusFarm dataset, KITTI, and Perseverance Navcam images.

How it works

  • Virtual stereo. Depth is decoded with the stereo relation, focal length times baseline over disparity. dRefNet corrects the backbone’s relative disparity per pixel (removing the unknown shift and local errors), and an MLP predicts one effective baseline scalar that carries the metric scale. Passing the known focal length in keeps camera intrinsics separate from scene scale.
  • Supervision. Masked L1 against projected sparse LiDAR points; an optional photometric loss that warps earlier frames using robot odometry, corrected by a small pose network (odomNet); and a gradient-matching loss that keeps the refined disparity’s structure aligned with the backbone’s.
  • Online loop. Images are sampled at 0.75 Hz at 220 px; each 10 s batch trains the next model, warm-started from the previous one.

Results

  • Against online-tuned metric DAV2, error within 10 m is 32% lower on average across garage, CitrusFarm and KITTI, with 16% less training time. Garage δ1 rises from 0.859 to 0.910; on CitrusFarm its RMSE is slightly worse (4.25 vs 4.01).
  • With only 3% of LiDAR points, garage error over 0-80 m grows from 1.28 to 1.43 m; with a single beam plus odomNet, δ1 is 0.765.
  • Replacing dense refinement by a least-squares scale-and-shift fit drops garage δ1 from 0.822 to 0.736.
  • Perseverance Navcam, with FoundationStereo as pseudo ground truth: δ1 0.975 vs 0.931 for online DAV2.

Limitations

The photometric loss assumes a static, Lambertian scene, so moving objects or occlusions corrupt training. With very sparse LiDAR and noisy wheel odometry, odomNet can add wrong pose corrections. The sampling rate must be tuned per robot, and frames from closely spaced positions can lower accuracy.

Figures

From Figure 1: Onboard metric depth adaptation on a Spot robot in a garage. Sparse image frames and matching point-cl...
From Figure 1: Onboard metric depth adaptation on a Spot robot in a garage. Sparse image frames and matching point-cloud views collected during exploration are used to refine a relative depth map into metric depth for the new environment.
From Figure 2: The virtual stereo system. From the backbone's relative inverse depth and the focal length, dRefNet pr...
From Figure 2: The virtual stereo system. From the backbone's relative inverse depth and the focal length, dRefNet predicts a refined dense disparity and an effective baseline. Earlier frames and their poses, corrected by odomNet, add photometric supervision alongside sparse LiDAR and a gradient-matching loss.
From Figure 5: Metric depth predicted during online learning on garage, CitrusFarm and KITTI images by unadapted DAV2...
From Figure 5: Metric depth predicted during online learning on garage, CitrusFarm and KITTI images by unadapted DAV2, online-tuned DAV2 and EM-ViT, with point maps of selected scenes shown against LiDAR ground truth (red).
Abstract

The geometric understanding of a scene is a crucial component for safe and efficient robotic operations in novel and unstructured environments. The recent emergence of foundation models for Monocular Depth Estimation (MDE) has provided an opportunity to enrich robotic perception capabilities with dense long-range depth predictions, replacing expensive onboard hardware (e.g., LiDARs and stereo cameras). However, these offline models are primarily trained on affine-invariant depth representations, and their generalization to novel scenes remains unclear. In this work we present EM-ViT, a monocular depth estimation model that learns to adapt online relative depth inputs from a foundation model prior to the metric scale with observations collected during robotic operations. Motivated by the stereo camera parameterization, our method decomposes depth prediction via dense disparity refinement to recover the scene's relative structure and an effective baseline scalar prediction to determine the missing metric scale. The learning process leverages sparse LiDAR points and, optionally, the robot's Ego-Motion (EM) to rapidly train a lightweight Vision Transformer (ViT), while an internal gradient matching loss preserves the fine-grained semantic structure of the input disparity in real-time. We demonstrate the efficacy of our method on publicly available sequences, such as KITTI, and show that our online learning pipeline can adapt to novel scenes faster and with higher accuracy than existing MDE foundation models. The code and supplementary materials will be publicly available at https://emvit-mde.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.