Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

★ Must read December 2015

TL;DR Let each few layers learn a residual F(x) that an identity shortcut adds back to the input, which makes very deep networks trainable. A 152-layer ResNet reaches 4.49% single-model top-5 error on ImageNet validation, and an ensemble gets 3.57% on the test set, 1st place in ILSVRC 2015.

What they do

Once networks with batch normalization start converging, adding layers past some depth makes them worse: on CIFAR-10 a 56-layer plain network has higher training error than a 20-layer one, so this is not overfitting. A deeper net could copy the shallow one and set its extra layers to identity, so the authors treat this “degradation” as an optimization problem. Each few stacked layers instead learn a residual F(x) = H(x) − x, and a shortcut adds x back. Representing an identity then only requires driving the weights toward zero.

How it works

Residual block. Two 3x3 conv layers compute F(x) and the block outputs ReLU(F(x) + x). Identity shortcuts add no parameters or computation; a 1x1 projection is used only where dimensions change.
Architecture. VGG-style rules: 3x3 filters, filters doubled when the feature map is halved, stride-2 convs for downsampling, global average pooling and one fully connected layer. The 34-layer net has 3.6 billion FLOPs, 18% of VGG-19.
Bottleneck blocks. The 50/101/152-layer nets use 1x1 (reduce), 3x3, 1x1 (restore) blocks. ResNet-152 needs 11.3 billion FLOPs, less than VGG-16/19 (15.3/19.6 billion).

Results

ImageNet: the 34-layer plain net is worse than the 18-layer one (28.54% vs 27.94% top-1), while the 34-layer ResNet beats both (25.03%). Error keeps falling with depth, down to 4.49% top-5 for ResNet-152; a six-model ensemble gets 3.57% on the test set.
CIFAR-10: a 110-layer ResNet reaches 6.43% error with 1.7M parameters; a 1202-layer net trains to under 0.1% training error.
Detection: swapping VGG-16 for ResNet-101 in Faster R-CNN raises COCO mAP@[.5, .95] from 21.2 to 27.2, a 28% relative gain.

Limitations

The 1202-layer CIFAR net tests worse than the 110-layer one (7.93% vs 6.43%), which the authors attribute to overfitting a small dataset without dropout or maxout. Why deep plain nets fail to optimize is left open: the authors rule out vanishing gradients and only conjecture exponentially slow convergence. The 110-layer net needed a learning-rate warm-up to start converging.

Figures

From Figure 1: Training error (left) and test error (right) on CIFAR-10 for 20-layer and 56-layer plain networks. The... — From Figure 1: Training error (left) and test error (right) on CIFAR-10 for 20-layer and 56-layer plain networks. The deeper network has higher training error, and thus higher test error.

From Figure 2: A residual building block. Two weight layers learn F(x), and an identity shortcut adds x to their outp... — From Figure 2: A residual building block. Two weight layers learn F(x), and an identity shortcut adds x to their output before the final ReLU.

From Figure 4: Training on ImageNet; thin curves are training error, bold curves validation error. Left: plain networ... — From Figure 4: Training on ImageNet; thin curves are training error, bold curves validation error. Left: plain networks of 18 and 34 layers. Right: ResNets of 18 and 34 layers, with no extra parameters over the plain ones.

Abstract

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong