Learning Transferable Visual Models From Natural Language Supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

★ Must read February 2021

TL;DR Train an image encoder and a text encoder on 400 million web (image, text) pairs to tell which caption goes with which image, then classify any dataset zero-shot by embedding its class names. CLIP reaches 76.2% zero-shot accuracy on ImageNet, matching the original ResNet-50 without using any of its labeled training images.

What they do

Standard vision models predict a fixed set of labeled categories. CLIP instead learns from 400 million (image, text) pairs collected from the internet, trained only to tell which caption goes with which image. Its text encoder then turns any dataset’s class names into a classifier, with no training on that dataset.

How it works

Contrastive objective. For a batch of 32,768 pairs, an image encoder and a text encoder maximize the cosine similarity of the true pairs and minimize it for all wrong pairings, with a symmetric cross-entropy loss and a learned temperature. This was 4x more efficient than predicting the caption’s bag-of-words, itself 3x faster than a transformer caption model.
Models. Five ResNets and three Vision Transformers trained from scratch for 32 epochs; the best is ViT-L/14 at 336 px.
Zero-shot classification. Each class name goes into a prompt such as “A photo of a {label}.”, and the image takes the class with the most similar embedding. Averaging 80 prompts adds 3.5% on ImageNet on top of 1.3% from the single prompt.

Results

76.2% zero-shot on ImageNet (95% top-5), up from 11.5% for Visual N-Grams.
Zero-shot CLIP beats a supervised linear classifier on ResNet-50 features on 16 of 27 datasets, and matches 4-shot logistic regression on its own features.
As a linear-probe backbone, the best CLIP beats the best existing model by 2.6% averaged over 12 datasets and 5% over 27.
Zero-shot models shrink the gap between ImageNet and natural distribution-shift accuracy by up to 75%. Fitting a classifier on ImageNet raises ImageNet accuracy by 9.2% to 85.4%, but average accuracy under shift slightly drops.

Limitations

Zero-shot CLIP is weak at fine-grained classification, counting and tasks unlikely to be in its data (distance to the nearest car is near chance), and gets 88% on MNIST, below logistic regression on raw pixels. The authors estimate about 1000x more compute to reach state of the art zero-shot. It can only choose among given classes, few-shot linear probes can underperform zero-shot, and it learns social biases from uncurated web data.

Figures

From Figure 1: CLIP jointly trains an image encoder and a text encoder to predict the correct pairings in a batch of ... — From Figure 1: CLIP jointly trains an image encoder and a text encoder to predict the correct pairings in a batch of (image, text) examples. At test time the text encoder embeds the target dataset's class names to build a zero-shot linear classifier.

From Figure 2: Zero-shot ImageNet accuracy against images processed. A transformer caption model learns 3x slower tha... — From Figure 2: Zero-shot ImageNet accuracy against images processed. A transformer caption model learns 3x slower than a baseline that predicts a bag-of-words encoding of the text, and swapping in CLIP's contrastive objective improves efficiency another 4x.

From Figure 5: Zero-shot CLIP against a fully supervised linear classifier on ResNet-50 features, across 27 datasets.... — From Figure 5: Zero-shot CLIP against a fully supervised linear classifier on ResNet-50 features, across 27 datasets. CLIP wins on 16, including ImageNet.

From Figure 13: Distribution shift for bananas, a class shared across 5 of the 7 natural distribution shift datasets.... — From Figure 13: Distribution shift for bananas, a class shared across 5 of the 7 natural distribution shift datasets. The best zero-shot CLIP model, ViT-L/14@336px, is compared with a ResNet-101 that has the same accuracy on the ImageNet validation set.

Abstract

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong