Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun

Seminal June 2015

TL;DR A Region Proposal Network shares convolutional features with the Fast R-CNN detector, so object proposals cost about 10 ms instead of seconds. With VGG-16 the whole detector runs at 5 fps and reaches 73.2% mAP on PASCAL VOC 2007 using 300 proposals per image.

What they do

By 2015, region-based detectors like Fast R-CNN ran in a fraction of a second on a GPU, but the proposals they relied on came from CPU methods: Selective Search at about 2 s per image, EdgeBoxes at 0.2 s. Faster R-CNN replaces these with a Region Proposal Network (RPN) built on the detector’s own convolutional feature map. Proposals then add only about 10 ms per image.

How it works

  • RPN. A 3x3 conv window slides over the last shared feature map. At each position, two sibling 1x1 layers output objectness scores and box offsets for k = 9 anchors (3 scales × 3 aspect ratios).
  • Anchors as regression references. Scale is handled by a “pyramid of anchors” on a single-scale image, with no image or filter pyramids. The anchors are also translation invariant.
  • Training. Each mini-batch samples 256 anchors from one image, up to half of them positive (IoU > 0.7 positive, < 0.3 negative). The loss is a classification term plus a smooth-L1 box term.
  • Feature sharing. A 4-step alternating scheme (RPN, then Fast R-CNN, then each refined on frozen shared layers) makes the two modules share one set of conv layers.

Results

  • PASCAL VOC 2007 (VGG-16). 69.9% mAP with 300 RPN proposals vs. 66.9% for Selective Search with 2000. That rises to 73.2% with VOC07+12 training data and 78.8% when COCO data is added.
  • Speed (K40 GPU). 198 ms per image in total (5 fps), vs. 1830 ms for Selective Search + Fast R-CNN. The ZF model runs at 17 fps.
  • Ablations. A single anchor costs 3-4 mAP. With the RPN’s box regression removed, mAP drops to 52.1%. The two-stage design beats a dense one-stage variant, 58.7% vs. 53.9%.
  • COCO test-dev. 42.1% mAP@0.5 and 21.5% mAP@[.5, .95], 2.8 and 2.2 points above Fast R-CNN.

Limitations

The alternating training scheme is a pragmatic workaround. Approximate joint training ignores gradients with respect to the proposal boxes, and exact joint training is left out of scope. Detection runs at a single input scale with a 16-pixel stride, and the authors note that accuracy may improve with a smaller stride. Anchors that cross the image boundary have to be dropped during training, or training does not converge.

Figures

From Figure 2: Faster R-CNN as one network. Shared conv layers feed both the Region Proposal Network and, through RoI...
From Figure 2: Faster R-CNN as one network. Shared conv layers feed both the Region Proposal Network and, through RoI pooling, the classifier; the RPN tells the detector where to look.
From Figure 1: Three ways to handle multiple scales: (a) pyramids of images and feature maps, (b) pyramids of filters...
From Figure 1: Three ways to handle multiple scales: (a) pyramids of images and feature maps, (b) pyramids of filters on one feature map, (c) the paper's pyramid of reference (anchor) boxes.
From Figure 3: Left, the RPN slides a small network over the conv feature map and predicts 2k objectness scores and 4...
From Figure 3: Left, the RPN slides a small network over the conv feature map and predicts 2k objectness scores and 4k box coordinates for k anchors. Right, example detections on PASCAL VOC 2007 test across many scales and aspect ratios.
From Figure 4: Proposal recall vs. IoU on PASCAL VOC 2007 test for 300, 1000 and 2000 proposals, comparing Selective ...
From Figure 4: Proposal recall vs. IoU on PASCAL VOC 2007 test for 300, 1000 and 2000 proposals, comparing Selective Search, EdgeBoxes and RPN with ZF and VGG.
Abstract

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features---using the recently popular terminology of neural networks with 'attention' mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model, our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.