Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

CoTracker: It is Better to Track Together

Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, Christian Rupprecht

Seminal July 2023

Tracks on a horse and rider clearing a show-jumping fence.

TL;DR A transformer point tracker that tracks many 2D points in a video jointly, with attention between tracks, instead of one at a time. Trained only on synthetic Kubric, it raises Average Jaccard on TAP-Vid-DAVIS from 56.2 (TAPIR) to 62.2.

What they do

Point trackers such as PIPs++ and TAPIR follow each query point independently, even though points on the same object move together. CoTracker estimates many tracks at once with a transformer that attends across both time and tracks, so occluded points can be inferred from their visible neighbours. It runs online on short overlapping windows and is trained only on synthetic TAP-Vid-Kubric.

How it works

  • Token grid. Each track and frame gets a token built from position, visibility, appearance and RAFT-style multi-scale correlation features. The transformer is applied M times (4 in training, 6 at test), each time predicting updates to positions and track features.
  • Factorised attention with proxy tokens. Attention alternates over time and over tracks. For track attention, regular tracks cross-attend to K learned proxy tracks (64 works best) instead of to each other, which lowers the track-attention cost from O(N²) to O(NK+K²).
  • Windowed, unrolled training. Windows of 8 frames overlap by half; each window starts from the previous window’s predictions, and training unrolls this as a recurrent network.
  • Support points. When the user asks for only a few points, extra points on a global grid and a local grid around the target are tracked alongside them for context.

Results

  • TAP-Vid-DAVIS (“first” protocol): AJ 62.2 and position accuracy 75.7, against 56.2 and 70.0 for TAPIR. On RGB-Stacking AJ rises to 71.6, against 60.3 for TAPIR and 69.5 for OmniMotion, which optimises per video.
  • Removing cross-track attention on Dynamic Replica lowers occluded-point accuracy from 37.6 to 28.8 and visible-point accuracy from 68.9 to 62.4.
  • Turning off unrolled training costs 18 AJ points.
  • Within 80 GB, proxy tokens allow 7.4x more tracks (a 263×263 grid) and run 7x faster than full self-attention at that size.

Limitations

Training only on synthetic data hurts generalisation to scenes with reflections and shadows; CoTracker tends to track shadows along with the objects that cast them. It is slower than TAPIR. It is built for continuous video, so its lead over TAPIR is smaller on the discontinuous clips of TAP-Vid-Kinetics.

Videos

Tracks on a person vaulting over a railing.
Tracks on a motocross rider jumping toward the camera.

Figures

From Figure 2: Non-joint tracking (top) vs. joint tracking (bottom), with background points in cyan and foreground po...
From Figure 2: Non-joint tracking (top) vs. joint tracking (bottom), with background points in cyan and foreground points in magenta. Tracked independently, background points follow the moving object; tracked together, both sets stay correct.
From Figure 3: CoTracker architecture. Convolutional features are computed for every frame and processed in sliding w...
From Figure 3: CoTracker architecture. Convolutional features are computed for every frame and processed in sliding windows; track features are sampled from them at the starting point locations, which also initialize the estimated tracks.
From Figure 4: One sliding window with M updates. Each iteration updates the point tracks and track features; visibil...
From Figure 4: One sliding window with M updates. Each iteration updates the point tracks and track features; visibility is computed after the last update.
From Figure 6: Qualitative results. PIPs++ (top) leaves many points stuck on the front of the car, TAPIR (middle) mov...
From Figure 6: Qualitative results. PIPs++ (top) leaves many points stuck on the front of the car, TAPIR (middle) moves chaotically once points are occluded, and CoTracker (bottom) produces cleaner tracks.
Abstract

We introduce CoTracker, a transformer-based model that tracks a large number of 2D points in long video sequences. Differently from most existing approaches that track points independently, CoTracker tracks them jointly, accounting for their dependencies. We show that joint tracking significantly improves tracking accuracy and robustness, and allows CoTracker to track occluded points and points outside of the camera view. We also introduce several innovations for this class of trackers, including using token proxies that significantly improve memory efficiency and allow CoTracker to track 70k points jointly and simultaneously at inference on a single GPU. CoTracker is an online algorithm that operates causally on short windows. However, it is trained utilizing unrolled windows as a recurrent network, maintaining tracks for long periods of time even when points are occluded or leave the field of view. Quantitatively, CoTracker substantially outperforms prior trackers on standard point-tracking benchmarks.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.