FAST: Efficient Action Tokenization for Vision-Language-Action Models

Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, Sergey Levine

Seminal January 2025

Autonomous table bussing with a UR5e arm, sped up 4x.

TL;DR Compress each 1-second robot action chunk with a discrete cosine transform and byte-pair encoding before training an autoregressive VLA on it. With these FAST tokens, π0 trained on 10k hours of robot data matches the diffusion version of π0 while needing 5x fewer GPU hours to train.

What they do

Autoregressive VLAs such as RT-2 and OpenVLA turn each action dimension at each timestep into one of 256 bins. At high control frequencies consecutive tokens are nearly identical, so each new token carries little information and models can drift toward copying the last action. In a toy spline-prediction task, error with binning rises steeply as the sampling rate grows. FAST compresses the action chunk first, so the model predicts fewer, more informative tokens.

How it works

DCT per dimension. Actions are normalized so the 1st and 99th percentiles map to [-1, 1], then each dimension of a 1-second chunk is transformed with the discrete cosine transform.
Quantize and flatten. Coefficients are scaled and rounded (scale 10), which zeroes most of them. The matrix is flattened lowest frequencies first across all dimensions, so the overall shape of the motion is predicted first.
BPE. Byte-pair encoding (vocabulary 1024) squashes the zeros into dense tokens that replace the least-used tokens of the VLM vocabulary. Every step is invertible for decoding.
FAST+. A universal tokenizer trained on about one million 1-second chunks from single-arm, bimanual and mobile robots with joint, end-effector and camera-frame actions.

Results

Tokens per chunk drop from 700 to 53 on T-shirt folding (50 Hz) and from 140 to 28 on table bussing (20 Hz), about 30 per arm regardless of frequency. FAST+ cuts token counts by 2x or more on 14 held-out datasets, from dexterous hands to humanoids and driving.
With binning, π0 makes no progress on table bussing or T-shirt folding. FAST is as good as or better than a learned FSQ tokenizer, and FAST+ matches per-dataset FAST. FAST+ also lifts OpenVLA on T-shirt folding.
Trained on DROID, the policy works zero-shot from language prompts in unseen scenes on three campuses.
π0-FAST on the π0 mixture (903M timesteps of the authors’ data plus open datasets) matches diffusion π0, including on laundry folding, with 5x fewer GPU hours.

Limitations

Inference takes about 750 ms per chunk on an RTX 4090 against about 100 ms for diffusion π0, since 30-60 tokens are decoded with the full 2B backbone. That was acceptable on the static tasks tested; the authors note faster decoding is needed for highly dynamic tasks. Policies ran only on static manipulators; FAST+ on mobile robots, hands and humanoids was checked for compression only. Rounding makes the tokens lossy.

Figures

From Figure 2: Left, FAST tokenizes high-frequency robot actions so a VLA can be trained with plain next-token predic... — From Figure 2: Left, FAST tokenizes high-frequency robot actions so a VLA can be trained with plain next-token prediction. Right, FAST outperforms the OpenVLA-style binning tokenizer, particularly on high-frequency data.

From Figure 4: The FAST pipeline. A normalized action chunk is converted to the frequency domain with the discrete co... — From Figure 4: The FAST pipeline. A normalized action chunk is converted to the frequency domain with the discrete cosine transform, the coefficients are quantized and flattened low-frequency first, and byte-pair encoding compresses them into action tokens.

From Figure 1: Top, evaluation score over training iterations: π0 with FAST tokens reaches the performance of the dif... — From Figure 1: Top, evaluation score over training iterations: π0 with FAST tokens reaches the performance of the diffusion π0 VLA with 5x less training. Bottom, example dexterous manipulation and generalization tasks.

Abstract

Autoregressive sequence models, such as Transformer-based vision-language action (VLA) policies, can be tremendously effective for capturing complex and generalizable robotic behaviors. However, such models require us to choose a tokenization of our continuous action signals, which determines how the discrete symbols predicted by the model map to continuous robot actions. We find that current approaches for robot action tokenization, based on simple per-dimension, per-timestep binning schemes, typically perform poorly when learning dexterous skills from high-frequency robot data. To address this challenge, we propose a new compression-based tokenization scheme for robot actions, based on the discrete cosine transform. Our tokenization approach, Frequency-space Action Sequence Tokenization (FAST), enables us to train autoregressive VLAs for highly dexterous and high-frequency tasks where standard discretization methods fail completely. Based on FAST, we release FAST+, a universal robot action tokenizer, trained on 1M real robot action trajectories. It can be used as a black-box tokenizer for a wide range of robot action sequences, with diverse action spaces and control frequencies. Finally, we show that, when combined with the pi0 VLA, our method can scale to training on 10k hours of robot data and match the performance of diffusion VLAs, while reducing training time by up to 5x.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong