Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer

René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, Vladlen Koltun

Seminal July 2019

TL;DR Train one monocular depth network on five datasets with incompatible ground truth, including a new dataset mined from 3D movies, using a scale- and shift-invariant loss and Pareto-optimal dataset mixing. Tested zero-shot on six unseen datasets, every prior method is at least 65% worse on average.

What they do

Depth datasets disagree in form: metric depth, depth up to scale (SfM), or disparity up to scale and shift (stereo with unknown calibration). This paper trains one monocular depth network on five such datasets, including a new one from 23 3D films, and tests it only on datasets never seen in training.

How it works

3D Movies dataset. Stereo frames from Blu-rays; the horizontal component of optical flow serves as disparity, since film disparities can be negative. A left-right consistency check and automatic filters remove bad frames, leaving 75,074 training frames from 19 films.
Loss. The network predicts disparity (inverse depth up to scale and shift). Prediction and ground truth are aligned in scale and shift per image, then compared with a trimmed absolute error that discards the 20% largest residuals, plus a multi-scale gradient-matching term.
Mixing. Each dataset is treated as a task, and training seeks a Pareto optimum across them, instead of sampling datasets in equal parts per minibatch.
Encoder. ResNeXt-101 pretrained on weakly-supervised data, then ImageNet.

Results

The trimmed loss gives the lowest validation error of all losses tested. The best encoder improves up to 15% over ResNet-50; a randomly initialized ResNet-50 is on average 35% worse than a pretrained one.
Each extra dataset used alone generalizes worse on average than ReDWeb alone. Mixing all five improves the mean by 19.5% over ReDWeb with naive mixing and 22.4% with Pareto mixing.
Zero-shot, the final model reaches WHDR 12.46 on DIW, AbsRel 0.129 on ETH3D and 0.327 on Sintel. The three best prior zero-shot models are 65-69% worse on average; a ResNet-50 version is 16% worse but still beats them.

Limitations

Depth is relative: errors are measured after aligning scale and shift to ground truth per image. The model learned that lower image regions are closer, which breaks on an image rotated by 90 degrees. It predicts the depicted content for paintings, photos and mirrors, hallucinates discontinuities at strong edges, misses thin structures and blurs the background.

Videos

Figures

From Figure 1: Inverse depth map predicted for a Microsoft COCO image; COCO was not seen during training.

From Figure 2: Ground-truth inverse depth map from a training film in the 3D Movies dataset, with sky regions and inv... — From Figure 2: Ground-truth inverse depth map from a training film in the 3D Movies dataset, with sky regions and invalid pixels masked out.

From Figure 3: Relative performance of loss functions (higher is better), with the trimmed MAE plus gradient loss as ... — From Figure 3: Relative performance of loss functions (higher is better), with the trimmed MAE plus gradient loss as reference. All four proposed losses (white area) outperform the prior losses (gray area).

From Figure 4: Relative performance of encoders against a ResNet-50 baseline (higher is better). ImageNet performance... — From Figure 4: Relative performance of encoders against a ResNet-50 baseline (higher is better). ImageNet performance of an encoder predicts its depth estimation performance.

Abstract

The success of monocular depth estimation relies on large and diverse training sets. Due to the challenges associated with acquiring dense ground-truth depth across different environments at scale, a number of datasets with distinct characteristics and biases have emerged. We develop tools that enable mixing multiple datasets during training, even if their annotations are incompatible. In particular, we propose a robust training objective that is invariant to changes in depth range and scale, advocate the use of principled multi-objective learning to combine data from different sources, and highlight the importance of pretraining encoders on auxiliary tasks. Armed with these tools, we experiment with five diverse training datasets, including a new, massive data source: 3D films. To demonstrate the generalization power of our approach we use zero-shot cross-dataset transfer, i.e. we evaluate on datasets that were not seen during training. The experiments confirm that mixing data from complementary sources greatly improves monocular depth estimation. Our approach clearly outperforms competing methods across diverse datasets, setting a new state of the art for monocular depth estimation. Some results are shown in the supplementary video at https://youtu.be/D46FzVyL9I8

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong