Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

Zachary Teed, Jia Deng

Seminal March 2020

TL;DR RAFT estimates optical flow by correlating every pixel with every other pixel and refining a single high-resolution flow field with a small weight-tied recurrent unit. It cuts the best published error by 30% on Sintel final (2.855 px EPE) and by 16% on KITTI (5.10% F1-all).

What they do

Most deep optical-flow networks work coarse-to-fine, so coarse errors are hard to undo and small fast-moving objects get missed. RAFT keeps one flow field at 1/8 resolution and refines it many times with the same small recurrent unit, which reads a correlation volume between all pairs of pixels.

How it works

  • Features. A residual encoder maps both images to 256-dim features at 1/8 resolution. A context encoder runs on the first image only.
  • All-pairs correlation. One matrix multiply correlates every feature pair. Pooling the second image’s dimensions with kernels 1, 2, 4 and 8 gives a 4-level pyramid. Each step bilinearly samples a radius-4 grid around every pixel’s current match on all levels.
  • Update operator. A convolutional GRU, with weights tied across iterations, turns correlation, flow and context features into a flow update. Flow starts at zero. Training unrolls 12 updates.
  • Upsampling and loss. Full-resolution flow is a learned convex combination of 3x3 coarse neighbours. The L1 loss covers every iterate, weighted toward later ones.

Results

  • Trained only on synthetic data, it reaches 1.43 EPE on Sintel clean (29% below FlowNet2) and cuts KITTI-15 EPE from 8.36 to 5.04.
  • It ranks first on both Sintel test passes (1.61 clean, 2.86 final EPE with warm-start) and on KITTI (5.10% F1-all against 6.10%).
  • 5.3M parameters, 0.10 s per 1088x436 pair on a 1080Ti, and 200k training iterations against FlowNet2’s 7000k. The 1.0M-parameter RAFT-S still beats prior methods on Sintel final (3.37 EPE).
  • Ablations: untied weights are less accurate and need 32.5M parameters instead of 4.8M. Warping instead of correlation raises KITTI EPE from 5.54 to 11.83.

Limitations

The paper does not discuss limitations. All-pairs correlation grows with the square of the pixel count. An on-demand alternative is described but not needed: at 1080p, correlation takes 17% of inference time (550 ms for 12 updates). Evaluation covers only Sintel and KITTI, plus qualitative DAVIS video.

Figures

From Figure 1: A feature encoder runs on both images and a context encoder on the first. The inner product of all fea...
From Figure 1: A feature encoder runs on both images and a context encoder on the first. The inner product of all feature pairs gives a 4D correlation volume, pooled at several scales, and a recurrent update operator uses the current flow to look up values in these volumes and update the flow.
From Figure 2: 2D slices of the 4D correlation volume. Each feature vector in the first image is correlated with all ...
From Figure 2: 2D slices of the 4D correlation volume. Each feature vector in the first image is correlated with all pixels of the second, and the volume is average-pooled with kernel sizes 1, 2, 4 and 8.
From Figure 3: Flow predictions on Sintel, shown next to ground truth, VCN and IRR-PWC.
From Figure 3: Flow predictions on Sintel, shown next to ground truth, VCN and IRR-PWC.
From Figure 5: Sintel (train) final-pass error after training on FlyingChairs and FlyingThings, plotted against param...
From Figure 5: Sintel (train) final-pass error after training on FlyingChairs and FlyingThings, plotted against parameter count (left), inference time (middle) and training iterations (right).
Abstract

We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance. On KITTI, RAFT achieves an F1-all error of 5.10%, a 16% error reduction from the best published result (6.10%). On Sintel (final pass), RAFT obtains an end-point-error of 2.855 pixels, a 30% error reduction from the best published result (4.098 pixels). In addition, RAFT has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count. Code is available at https://github.com/princeton-vl/RAFT.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.