ActionCodec2: A Streaming Action Tokenizer

Zibin Dong, Yicheng Liu, Shiduo Zhang, Tianyuan Yuan, Baijun Ye, Jinqiu Wang, Hang Zhao

November 2026

Real Franka, zero-shot snack storing and serving: ActionCodec2 vs FAST vs X-Tokenizer, 5x speed.

TL;DR A streaming action tokenizer for autoregressive VLAs that quantizes each control step against the decoder's reconstruction, so trajectory error stays within a set tolerance, at 8.8 tokens per second. In the same VLA it reaches 73.9% task progress zero-shot on a real Franka, against 50.8% for FAST, with 50 ms latency instead of 607 ms.

What they do

Most action tokenizers for autoregressive VLAs encode one to two seconds of motion as a chunk that must be fully decoded before any action runs, and small per-step errors add up: in LIBERO replays, FAST reconstructs single actions within 1 mm but drifts about 40 mm per episode. ActionCodec2 is a streaming tokenizer, fitted without gradient training, that releases each control step as soon as its tokens are out while bounding the trajectory error.

How it works

Error-feedback quantization. The encoder mirrors the decoder’s reconstructed state and picks the fixed motion primitive that returns the tracking error inside the tolerance (5 mm, 0.02 rad). Primitives only cover one step of motion: 17 per axis at 1.2 m/s and 15 Hz, against 400 bins for absolute positions in a 4 m workspace. They derive the exact minimum for translation and constant-factor-tight bounds for rotation.
Set-BPE. Losslessly merges co-occurring atoms across coordinates and neighboring steps with no fixed coordinate order, using 6.6-10.7% fewer tokens than BPE.
Universal version. Fitted on about 20k hours from 24 embodiments with 4,096 tokens: 8.8 tokens/s for end-effector control at 15 Hz.

Results

With the same G0.5 VLA: 98.2% on LIBERO, 93.8% on SimplerEnv and 79.5% on LIBERO-Plus, above FAST, OAT, X-Tokenizer and ActionCodec.
Real Franka, trained on DROID and tested zero-shot in unseen scenes: 73.9% task progress vs 50.8% for FAST and 52.8% for X-Tokenizer, with 50 ms latency vs 607 and 467 ms (one control period is 66.7 ms).
Under a 20 Hz real-time clock, streaming execution reaches 91.0% on LIBERO vs 73.1% when the same predictions wait for the full chunk.
Feeding the measured state back within a chunk raises success from 64.7% to 79.0% under randomized physics in simulation.

Limitations

The pretrained tokenizers cover end-effector speeds up to about 1.2 m/s, and on highly dynamic motion their compression may fall below that of chunk tokenizers. Real-robot tests use one Franka arm and four task suites.

Videos

Real Franka, zero-shot nested-cup arrangement: ActionCodec2 vs FAST vs X-Tokenizer, 5x speed.

In simulation, a closed-loop policy grasps an object moving along a random path.

Figures

From Figure 1: Left: a chunk tokenizer releases actions only after the whole chunk is decoded, while a streaming toke... — From Figure 1: Left: a chunk tokenizer releases actions only after the whole chunk is decoded, while a streaming tokenizer releases each action with its own tokens, so the policy can use feedback at every step. Right: a fixed VQ codebook places its codes by the training distribution, while ActionCodec2 covers the reachable range.

From Figure 3: (a) Each step of an action is quantized into primitive atoms, one per coordinate. (b) Set-BPE merges a... — From Figure 3: (a) Each step of an action is quantized into primitive atoms, one per coordinate. (b) Set-BPE merges atoms that co-occur within a step and across nearby steps, in any coordinate order. (c) The VLA emits tokens autoregressively and releases each action once its step is complete. (d) The decoded primitives track the trajectory within the tolerance.

From Figure 6: Real-robot task progress on four DROID-Franka task suites, 50 rollouts per policy, for policies traine... — From Figure 6: Real-robot task progress on four DROID-Franka task suites, 50 rollouts per policy, for policies trained on DROID and tested zero-shot in unseen scenes. The right panel gives each policy's inference latency.

From Figure 7: Example ActionCodec2 rollouts on the real DROID-Franka tasks.

Abstract

Most action tokenizers for autoregressive vision language action (VLA) models compress motion into chunks that must be decoded in full before their actions become executable. We introduce ActionCodec2, a non-neural streaming tokenizer that releases actions incrementally while keeping trajectory reconstruction error within a prescribed tolerance. Its encoder quantizes each step relative to the decoder's reconstructed state, so the selected motion primitive corrects residual error rather than allowing errors from independently quantized increments to accumulate. Finding the smallest such primitive set is a geometric covering problem; we derive the exact minimum for translation and constant-factor-tight bounds for rotation. Set-BPE then losslessly merges co-occurring primitive symbols across coordinates and neighboring steps, without a fixed coordinate ordering. A universal version fitted on 20k hours of data from 24 embodiments averages 8.8 tokens per second for 15 Hz end-effector motion on this corpus. With the same VLA backbone, ActionCodec2 outperforms the compared tokenizers on three simulation benchmarks and in zero-shot real-robot evaluation. Its streaming representation allows policies to incorporate state feedback within an action chunk, substantially improving performance over open-loop execution under randomized dynamics, external disturbances, and moving objects. The same tokens also can serve as a direct control interface for VLM agents, which compose successful action sequences into reusable macros. These agent-built macros reduce control decisions and thinking-token usage across attempts and transfer to a held-out task. More results are available at https://zibindong.github.io/actioncodec2-page/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong