ActionCodec2
A Streaming Action Tokenizer
Today’s VLAs act in chunks: plan a second, then go blind. ActionCodec2 gives every control step its own tokens — no neural network, a guaranteed error bound at every step.
Skip to the three-line quickstart ↓
A Streaming Action Tokenizer
Today’s VLAs act in chunks: plan a second, then go blind. ActionCodec2 gives every control step its own tokens — no neural network, a guaranteed error bound at every step.
Skip to the three-line quickstart ↓
Part I
What if every control step had its own tokens?
01Why streaming
Encodes 1–2 s of motion at once. The robot waits for the whole chunk, then runs it open-loop.
Gives every control step its own tokens. The robot moves as each token lands and can react at any step.
02Covering
Accurate at every step, even on motion never seen in training: the reconstruction stays within ε.
One step, one axis. Only the range reachable in one step needs cells.
true actiontokenized actionallowed error ±ε
The same step in rotation, with cells covering the ball of reachable turns.
true orientationtokenized orientationallowed error ±ε
How few tokens that takes: for a real arm, 400 bins per axis become 17.
per axis · arm range 4 m · ±5 mm tolerance · 15 actions per second · top speed 1.2 m/s
400 → 17bins per axis for a real arm — and 17 is provably the fewest that keep every step within ε
03Set-BPE
One second of motion starts as 7 tokens per step, 105 in all. Set-BPE merges tokens that occur together, as unordered sets — across axes, then across steps — down to about 9 on average. It is lossless, so the ε bound is untouched, and the stream stays causal: each step runs as soon as its tokens arrive.
One second of a real robot arm, into tokens and back
8.0–11.9%fewer tokens than order-dependent BPE, on held-out data
Why not learn a VQ codebook instead?
training datatokens per action: fewer → morean action never seen in training
04Universal
Everything above, fitted on data from 24 embodiments and released with 1k to 16k tokens at 1, 5 or 10 mm precision. The default, used in every experiment here, is 4k tokens at 5 mm. No version has weights to train or load, so any autoregressive VLA can adopt one as its action vocabulary.
20khours
Data from 24 embodiments
309M frames in all: 105M end-effector and 204M joint.
1k–16ktokens
Universal
1k, 2k, 4k, 8k or 16k tokens; the default is 4k. Out of distribution it may lose compression, never accuracy.
0trained weights
NN-free
Closed-form covering primitives and exact greedy Set-BPE. No network, no training.
8.8tokens/s
Compact
With the default 4k / 5 mm version, for 15 Hz end-effector motion (10.8 for joints). Set-BPE packs them losslessly, with 11.9% fewer tokens than order-dependent BPE.
1 / 5 / 10mm
Bounded tracking error
At every step of a rollout of any length. The default 5 mm version also bounds rotation to 0.02 rad.
1step at a time
Streaming
Tokens for every control step, so the policy reacts to each new observation instead of waiting for a chunk.
x−1o
Physical meaning
Every token names a motion: x−1o moves 1 cm left with the gripper open, so an agent can write tokens directly.
drop-in
Plug and play
The action vocabulary of any autoregressive VLA. The decoder is stateless, with no weights to load.
Try it
On Hugging Face
All versions on Hugging FaceThree lines to use it. Any robot, any autoregressive VLA.
import numpy as np
from transformers import AutoProcessor
codec = AutoProcessor.from_pretrained(
"ZibinDong/ActionCodec2-1st-order",
trust_remote_code=True,
)
tokens = codec.encode(actions, fps=15)
decoded_actions = codec.decode(tokens, fps=15)Part II
A streaming action tokenizer: does it work?
05Benchmarks
One VLA (G0.5), one training recipe; only the action tokenizer changes.
Four benchmarks: ActionCodec2 variants take the top places on all four. Axes are zoomed to where the methods differ.
On the real robot: zero-shot DROID-Franka rollouts, side by side. Every tokenizer is off the shelf, except X-Tokenizer, which is also fine-tuned on DROID.
73.9%average task progress for ACodec2+ across the four suites, vs 50.8 / 52.8 for FAST and X-Tokenizer — and it finishes visibly faster
06RL fine-tuning
PPO on a harder LIBERO-Object: objects and basket re-placed every episode. From about 5% success, ActionCodec2 variants climb to 83–93%. Among the baselines, ACodec comes closest at 91%; FAST, X-Tokenizer and OATaligned level off at 72–78%, and OATofficial never takes off.
Dashed buttons: learning curve only, no rollout clips.
Rolling mean over PPO steps, band ± one rolling std. Click the plot to jump to step 0, 100 or 200.
07In-chunk feedback
ActionCodec2 decodes step by step, so new observations can enter while a chunk is still being generated. Closed-loop ACodec2+ also reads the measured robot or object state after every executed action; open-loop ACodec2+ and the baselines see it only at the start of each chunk. Three LIBERO suites disturb the robot or the objects in the middle of a chunk, and every policy is trained there with PPO.
Every object and the basket wander across the table at 0.18–0.30 m/s and turn every 0.25–0.45 s, so the target has moved by the time a chunk ends.
Grasp rate averages the ten tasks, 50 fixed seeds each, after 200 PPO iterations; the clips come from the same checkpoints. In these clips, open-loop and FAST execute all 20 actions of each chunk.
Each episode draws payload mass, joint damping and friction, and surface friction from the hard ranges, so the arm falls short of its commanded motion. The clips show the same two seeds for all three policies: a heavy bowl to lift onto the cabinet and a plate to push to the stove.
Task success averages the six tasks, 50 fixed seeds each, after 400 PPO iterations. Clips come from the latest checkpoint saved when they were recorded: 400 (closed-loop), 350 (open-loop) and 250 (FAST) PPO iterations.
Once per second, a 150–180 N push hits the gripper in a random direction, knocking it off its path in the middle of a chunk.
Grasp rate averages the ten tasks, 50 fixed seeds each, after 150 PPO iterations. In these clips, open-loop and FAST execute all 20 actions of each chunk.
46.2%grasp rate under random motion for closed-loop ACodec2+, vs 27.6% open-loop and at most 18.0% for any baseline
08Agents
What if a VLM agent could write action tokens itself — and pack what works into reusable macros? Every token is a fixed motion with a readable name: x−1o moves 1 cm left, gripper open.
Look up, act, reflect, grow the library
Retained macros beat pose commands. With a library that carries over, decisions per submission fall from 17.0 to 8.3 over four rounds, against about 45 for language commands with absolute pose targets.
What the agent learned: 13 macros, each named and described in its own words.
Pick any of the 13 macros
The macros carry over. On a new task: 63 decisions and a failure from scratch, 16 and a success with the macros.
The same new task, run twice
Move the cube between the mats onto the empty mat, press the button once, then move the other cube into the gap.
One segment per decision; orange segments call a macro.
Thinking
Output
A long-horizon case study, from scratch: 20 episodes of Multi-Drawer Search on a bimanual robot. The agent explores and fails; each breakthrough becomes new macros. Pick a milestone to replay it.
Nine first-time milestones, replayed
Search two cabinets for a measuring cup and tongs; place each in the red bin, cup first.
Thinking
Macros
Citation