You Only Look Once: Unified, Real-Time Object Detection

Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi

Seminal June 2015

TL;DR YOLO treats object detection as a single regression, where one convolutional network sees the whole image once and outputs boxes and class probabilities on a 7x7 grid. It reaches 63.4% mAP on VOC 2007 at 45 fps, and the smaller Fast YOLO reaches 52.7% at 155 fps, twice the mAP of earlier real-time detectors.

What they do

Detectors such as DPM and R-CNN run a classifier over sliding windows or region proposals, with each stage trained separately, and are slow. YOLO replaces the pipeline with one convolutional network that maps pixels directly to boxes and class probabilities in a single pass, trained end-to-end on detection.

How it works

Grid output. The image is split into an S x S grid, and the cell holding an object’s center is responsible for it. Each cell predicts B boxes (x, y, w, h and a confidence equal to Pr(Object) x IOU) and one set of C class probabilities. On VOC this is a 7x7x30 tensor holding 98 boxes.
Network. 24 convolutional layers and 2 fully connected layers, the first 20 pretrained on ImageNet, at 448x448 input. Fast YOLO uses 9 convolutional layers.
Loss. Sum-squared error, with coordinate errors weighted up (5) and empty-cell confidence weighted down (0.5). Predicting the square root of width and height makes small-box errors count more.
Inference. One network pass per image. Non-max suppression adds 2-3% mAP.

Results

VOC 2007: YOLO gets 63.4% mAP at 45 fps on a Titan X and Fast YOLO 52.7% at 155 fps, against 26.1% for 30Hz DPM. Fast R-CNN reaches 70.0% at 0.5 fps.
Rescoring Fast R-CNN with YOLO raises mAP from 71.8% to 75.0%. Other Fast R-CNN ensembles add only 0.3-0.6%.
VOC 2012: YOLO scores 57.9%. Fast R-CNN + YOLO scores 70.7%, fourth on the leaderboard.
On the Picasso and People-Art artwork datasets, YOLO’s person AP drops less than R-CNN’s or DPM’s.

Limitations

Each cell predicts only two boxes and one class, so YOLO misses small objects in groups such as flocks of birds. It also struggles with unusual aspect ratios, and localization is its main source of error. On VOC 2012 it scores 8-10% below R-CNN or Feature Edit on bottle, sheep and tv/monitor.

Videos

Figures

From Figure 1: The YOLO pipeline: resize the image to 448x448, run a single convolutional network, and keep the detec... — From Figure 1: The YOLO pipeline: resize the image to 448x448, run a single convolutional network, and keep the detections the model is confident in.

From Figure 2: The image is split into an S x S grid; each cell predicts B boxes with confidences and C class probabi... — From Figure 2: The image is split into an S x S grid; each cell predicts B boxes with confidences and C class probabilities, which combine into the final detections.

From Figure 4: Error breakdown on VOC 2007. YOLO makes more localization errors than Fast R-CNN (19.0% vs 8.6%) but f... — From Figure 4: Error breakdown on VOC 2007. YOLO makes more localization errors than Fast R-CNN (19.0% vs 8.6%) but far fewer background false positives (4.75% vs 13.6%).

From Figure 6: YOLO on artwork and natural images from the internet. It is mostly correct, though it labels one perso... — From Figure 6: YOLO on artwork and natural images from the internet. It is mostly correct, though it labels one person an airplane.

Abstract

We present YOLO, a new approach to object detection. Prior work on object detection repurposes classifiers to perform detection. Instead, we frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. Our unified architecture is extremely fast. Our base YOLO model processes images in real-time at 45 frames per second. A smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors. Compared to state-of-the-art detection systems, YOLO makes more localization errors but is far less likely to predict false detections where nothing exists. Finally, YOLO learns very general representations of objects. It outperforms all other detection methods, including DPM and R-CNN, by a wide margin when generalizing from natural images to artwork on both the Picasso Dataset and the People-Art Dataset.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong