Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, Hengshuang Zhao

Seminal January 2024

Person in headphones; input video, MiDaS and Depth Anything depth, left to right.

TL;DR A monocular relative-depth model trained on 1.5M labeled images plus 62M unlabeled images pseudo-labeled by a teacher, with strong perturbations on the student and a feature-alignment loss to frozen DINOv2. Zero-shot on six unseen datasets it beats MiDaS v3.1 (KITTI AbsRel 0.076 vs 0.127), and fine-tuned it reaches 0.984 delta1 on NYUv2.

What they do

Depth Anything is a monocular relative-depth model meant to work on any image. Rather than new modules, it scales up data: a teacher trained on 1.5M labeled images from six public datasets pseudo-labels 62M unlabeled images from eight datasets (SA-1B, ImageNet-21K, BDD100K and others), and a student is retrained on both. Plain self-training on them did not beat the labeled-only baseline; two changes fix that.

How it works

Teacher. A DINOv2 encoder with a DPT decoder, trained like MiDaS with an affine-invariant loss in disparity space. NYUv2 and KITTI are left out of training so that evaluation on them stays zero-shot.
Harder target for the student. The teacher labels clean images, but the student sees them with strong color jitter, Gaussian blur and, half the time, CutMix. The student is re-initialized rather than fine-tuned from the teacher; each batch is one-third labeled, two-thirds unlabeled.
Semantic prior. An auxiliary segmentation head did not help, so instead a cosine feature-alignment loss keeps the student’s features close to a frozen DINOv2 encoder. Pixels already above a similarity margin of 0.85 are skipped, so parts of one object can still differ in depth.

Results

Zero-shot, ViT-L against MiDaS v3.1 ViT-L: KITTI AbsRel 0.076 vs 0.127 and DDAD 0.230 vs 0.251, even though MiDaS trained on KITTI and NYUv2 images. The ViT-B model already beats MiDaS on most metrics.
Ablation: adding pseudo-labeled images alone leaves KITTI AbsRel at 0.085; perturbations bring it to 0.081 and feature alignment to 0.076.
Fine-tuned for metric depth: NYUv2 delta1 0.984 and AbsRel 0.056 (previous best VPD: 0.964 and 0.069), KITTI delta1 0.982. Swapped into ZoeDepth, the encoder improves zero-shot metric depth on all five unseen datasets.
The encoder also transfers to segmentation: 86.2 mIoU on Cityscapes and 59.4 on ADE20K (previous best 58.3).

Limitations

The model outputs relative depth; zero-shot evaluation aligns scale and shift to ground truth, and metric depth needs fine-tuning (NYUv2 for indoor, KITTI for outdoor). The largest model is ViT-L, and the authors consider the roughly 512x512 training resolution too low for real-world applications.

Videos

Twirling ballerina; input, MiDaS and Depth Anything depth, left to right.

Roller-coaster track against the sky; input, MiDaS and Depth Anything depth, left to right.

Figures

From Figure 1: Zero-shot predictions on unseen images: COCO (left two columns), held-out SA-1B (middle two) and the a... — From Figure 1: Zero-shot predictions on unseen images: COCO (left two columns), held-out SA-1B (middle two) and the authors' own photos (right two), including low light, complex scenes, fog and very distant buildings.

From Figure 2: Pipeline. Labeled images (solid line) are supervised by manual labels; unlabeled images (dotted line) ... — From Figure 2: Pipeline. Labeled images (solid line) are supervised by manual labels; unlabeled images (dotted line) get strong perturbations S and are supervised by the teacher's pseudo labels, while an auxiliary loss to a frozen encoder preserves semantics.

From Figure 3: Qualitative results on the six unseen test datasets.

Abstract

This work presents Depth Anything, a highly practical solution for robust monocular depth estimation. Without pursuing novel technical modules, we aim to build a simple yet powerful foundation model dealing with any images under any circumstances. To this end, we scale up the dataset by designing a data engine to collect and automatically annotate large-scale unlabeled data (~62M), which significantly enlarges the data coverage and thus is able to reduce the generalization error. We investigate two simple yet effective strategies that make data scaling-up promising. First, a more challenging optimization target is created by leveraging data augmentation tools. It compels the model to actively seek extra visual knowledge and acquire robust representations. Second, an auxiliary supervision is developed to enforce the model to inherit rich semantic priors from pre-trained encoders. We evaluate its zero-shot capabilities extensively, including six public datasets and randomly captured photos. It demonstrates impressive generalization ability. Further, through fine-tuning it with metric depth information from NYUv2 and KITTI, new SOTAs are set. Our better depth model also results in a better depth-conditioned ControlNet. Our models are released at https://github.com/LiheYoung/Depth-Anything.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong