DINOv2: Learning Robust Visual Features without Supervision

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, Piotr Bojanowski

Seminal April 2023

TL;DR A 1.1B-parameter ViT trained with self-supervision on 142M automatically curated images, then distilled into smaller models. Its frozen features reach 86.5% ImageNet-1k linear-probe accuracy, slightly ahead of OpenCLIP, and beat prior self-supervised features on segmentation, depth and retrieval.

What they do

Self-supervised image features had mostly been trained on ImageNet-1k or on large uncurated web sets, which hurt feature quality. DINOv2 tests whether self-supervised pretraining on a large curated dataset yields general-purpose features that work frozen, without finetuning. The authors build a 142M-image dataset automatically, train a 1.1B-parameter ViT-g with a revised DINO/iBOT recipe, and distill it into ViT-S, B and L models.

How it works

Data (LVD-142M). From 1.2B crawled web images, near-duplicates (and copies of benchmark test images) are removed. Images are then retrieved by embedding similarity to curated datasets such as ImageNet-22k, Google Landmarks and fine-grained sets. No text or metadata is used.
Objective. A DINO loss on the class token plus an iBOT masked-patch loss, with separate heads, Sinkhorn-Knopp centering, a KoLeo regularizer that spreads features within a batch, and a short final phase at 518x518.
Efficiency. Custom FlashAttention, sequence packing, stochastic depth that skips dropped residuals, and FSDP: about 2x faster with 1/3 the memory of the iBOT code.
Distillation. Smaller models are distilled from the frozen ViT-g; a distilled ViT-L beats one trained from scratch on all 12 benchmarks tested.

Results

ImageNet-1k linear probe on frozen features: 86.5% for ViT-g, +4.2% over the previous best self-supervised model (iBOT) and +0.3% over OpenCLIP ViT-G/14.
Instance retrieval on Oxford-Hard: +41% mAP over self-supervised and +34% over weakly-supervised features.
Frozen dense features: 53.0 mIoU on ADE20k with a boosted linear head (OpenCLIP-G: 46.0); NYUd depth RMSE 0.279 with a DPT decoder (OpenCLIP-G: 0.414).
Curation matters: at equal iterations, LVD-142M beats uncurated data on most benchmarks (iNat2018 82.3 vs 68.0).

Limitations

DINOv2 still trails OpenCLIP on ImageNet-R, Sketch, Places205 (-2.3%), SUN (-5.3%) and Cars (-4.7%). The fairness study finds accuracy 25.7% lower for Africa than Europe and 31.7% lower for low- than high-income households. One ViT-g pretraining takes about 22k GPU-hours.

Figures

From Figure 1: First three PCA components of patch features, computed across the images of each column and mapped to ... — From Figure 1: First three PCA components of patch features, computed across the images of each column and mapped to color channels. The same parts match between related images despite changes of pose, style or object; background is removed by thresholding the first component.

From Figure 3: Data pipeline. Curated and uncurated images are embedded, uncurated images are deduplicated, then retr... — From Figure 3: Data pipeline. Curated and uncurated images are embedded, uncurated images are deduplicated, then retrieved by similarity to curated images to augment the curated set.

From Figure 2: Performance as models scale up, on eight types of vision tasks. DINOv2 (dark blue) is compared with se... — From Figure 2: Performance as models scale up, on eight types of vision tasks. DINOv2 (dark blue) is compared with self-supervised methods (pale orange) and weakly-supervised methods (dark pink); the dashed line is the best weakly-supervised model.

From Figure 7: Segmentation on ADE20K and depth on NYUd, SUN RGB-D and KITTI from a linear probe on frozen OpenCLIP-G... — From Figure 7: Segmentation on ADE20K and depth on NYUd, SUN RGB-D and KITTI from a linear probe on frozen OpenCLIP-G and DINOv2-g features.

Abstract

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong