Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang

Seminal March 2023

TL;DR Grounding DINO turns the Transformer detector DINO into an open-set detector that finds any object named in a text prompt, by fusing language and image features in the neck, the query initialization and the decoder. It reaches 52.5 AP on COCO without any COCO training data and a zero-shot record of 26.1 mean AP on ODinW.

What they do

Grounding DINO detects arbitrary objects named in text, either a list of category names or a referring expression with attributes. It adds language to the closed-set Transformer detector DINO and pretrains on detection, grounding and caption data, following GLIP’s formulation of detection as phrase grounding. The authors argue that fusing language at more stages of the detector helps open-set generalization, and evaluate on closed-set, zero-shot and referring-expression benchmarks.

How it works

Feature enhancer. A Swin Transformer gives multi-scale image features and BERT-base gives text features. Six layers fuse them with deformable self-attention on image tokens, self-attention on text tokens, and image-to-text and text-to-image cross-attention.
Language-guided query selection. The 900 image tokens most similar to the text initialize the decoder queries’ anchor boxes; the content queries stay learnable.
Cross-modality decoder. Six layers, each a DINO decoder layer plus a text cross-attention. Each query is dot-producted with every text token to classify it (focal loss), with L1 and GIoU box losses.
Sub-sentence prompts. When category names are concatenated into one prompt, attention masks stop unrelated names from attending to each other while keeping per-word features.

Results

COCO zero-shot: the Swin-T model gets 48.1 AP vs. 46.7 for GLIP-T on the same O365 and GoldG data; the Swin-L model gets 52.5 AP, and 63.0 AP on test-dev after COCO fine-tuning with larger images.
ODinW zero-shot: 26.1 mean AP, above Florence’s 25.8. Swin-T matches GLIPv2-T’s 22.3 mean AP with a higher median (11.9 vs. 8.9) and fewer parameters (172M vs. 232M).
LVIS zero-shot: 27.4 AP vs. 26.0 for GLIP-T, but lower on rare categories (18.1 vs. 20.8) and far below DetCLIPv2’s 40.4. On RefCOCO val, accuracy is 50.41 without referring data and 89.19 after fine-tuning.
Ablation (Swin-T, O365): removing encoder fusion drops LVIS zero-shot AP from 16.1 to 13.1; static query selection drops it to 13.6.

Limitations

Boxes only, no segmentation. Its training data is smaller than the largest GLIP model’s, and it produces false positives in some cases. Rare LVIS categories remain weak, which the authors suggest may be a limitation of DETR-like detectors, and zero-shot referring-expression accuracy is low without referring training data.

Figures

From Figure 1: (a) closed-set detection of pre-defined categories; (b) open-set detection of novel categories and ref... — From Figure 1: (a) closed-set detection of pre-defined categories; (b) open-set detection of novel categories and referring expressions such as "The left lion"; (c) image editing by combining Grounding DINO with Stable Diffusion.

From Figure 2: extending a closed-set detector (backbone, neck, head) to open-set detection by fusing text features a... — From Figure 2: extending a closed-set detector (backbone, neck, head) to open-set detection by fusing text features at the neck (A), the query initialization (B) and the head (C).

From Figure 3: the Grounding DINO framework (block 1), a feature enhancer layer (block 2) and a cross-modality decode... — From Figure 3: the Grounding DINO framework (block 1), a feature enhancer layer (block 2) and a cross-modality decoder layer (block 3).

From Figure 6: model detections on different scenes and text inputs.

Abstract

In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean $26.1$ AP. Code will be available at \url{https://github.com/IDEA-Research/GroundingDINO}.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong