Mask R-CNN

Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick

Seminal March 2017

TL;DR Add a mask-predicting branch and a quantization-free RoIAlign layer to Faster R-CNN to get instance segmentation at 5 fps. Mask R-CNN reaches 37.1 mask AP on COCO test-dev, above the 2016 challenge winner, and the same framework also detects boxes and human keypoints.

What they do

Mask R-CNN extends the two-stage Faster R-CNN detector to instance segmentation. Next to the existing class and box branches, each region of interest (RoI) gets a third branch: a small fully convolutional network that predicts a binary mask. With masks swapped for one-hot keypoint masks, the same framework also does human pose estimation.

How it works

Parallel heads. A region proposal network proposes boxes. For each RoI the second stage predicts a class, a box offset and an m×m mask, trained with the sum of the three losses.
Decoupled masks. The mask branch outputs one binary mask per class with a per-pixel sigmoid. Only the ground-truth class’s mask enters the loss, and at test time the classification branch picks which mask to use, so classes do not compete as they do under an FCN’s per-pixel softmax.
RoIAlign. RoIPool rounds RoI boundaries and bins to the feature grid, misaligning features with the image. RoIAlign drops all rounding and bilinearly samples four points per bin.
Keypoints. Each of K keypoints is a one-hot 56×56 mask, trained with a softmax over pixels.

Results

COCO test-dev instance segmentation: 37.1 mask AP with ResNeXt-101-FPN, versus 33.6 for FCIS+++, the 2016 winner, which adds multi-scale train/test, flip testing and OHEM.
Ablations on minival: per-class sigmoid beats softmax by 5.5 mask AP. RoIAlign beats RoIPool by about 3 AP on a stride-16 backbone and by 7.3 AP on stride 32. An FCN mask head beats an MLP by 2.1 AP.
Box detection: 39.8 box AP on test-dev. Training with the mask branch adds 0.9 box AP over the same model without it.
Keypoints: 63.1 keypoint AP on test-dev versus 61.8 for the 2016 winner, from one model that predicts boxes, masks and keypoints at 5 fps. ResNet-101-FPN runs at 195 ms per image on a Tesla M40 and trains in 44 h on 8 GPUs.

Limitations

The design is not optimized for speed, and the C4 variant takes about 400 ms per image. The mask head is deliberately simple, and pose estimation uses almost no domain knowledge. Adding the keypoint branch slightly lowers box and mask AP. On Cityscapes, the truck, bus and train classes (about 200-500 training samples each) show a val/test gap that COCO pre-training narrows but does not close.

Figures

From Figure 1: The Mask R-CNN framework for instance segmentation.

From Figure 3: RoIAlign. The dashed grid is a feature map, the solid lines an RoI with 2x2 bins, and the dots the 4 s... — From Figure 3: RoIAlign. The dashed grid is a feature map, the solid lines an RoI with 2x2 bins, and the dots the 4 sampling points in each bin. Each sampling point is bilinearly interpolated from nearby grid points, with no quantization of the RoI, its bins or the sampling points.

From Figure 2: Mask R-CNN results on the COCO test set with a ResNet-101 model (35.7 mask AP, 5 fps). Masks are shown... — From Figure 2: Mask R-CNN results on the COCO test set with a ResNet-101 model (35.7 mask AP, 5 fps). Masks are shown in color, along with boxes, categories and confidences.

From Figure 7: Keypoint detection on COCO test with Mask R-CNN (ResNet-50-FPN), with person masks predicted by the sa... — From Figure 7: Keypoint detection on COCO test with Mask R-CNN (ResNet-50-FPN), with person masks predicted by the same model (63.1 keypoint AP, 5 fps).

Abstract

We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition. Code has been made available at: https://github.com/facebookresearch/Detectron

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong