$π_0$: A Vision-Language-Action Flow Model for General Robot Control

Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, Ury Zhilinsky

★ Must read October 2024

A bimanual robot autonomously pulls a shirt from a laundry bin and folds it, 1x speed.

TL;DR π0 adds a flow-matching action expert to the PaliGemma vision-language model and pre-trains it on about 10,000 hours of data from 7 robot configurations plus open datasets, giving one 3.3B-parameter policy that controls robots at up to 50 Hz. Without fine-tuning it beats OpenVLA and Octo on every tested task, and after fine-tuning it folds laundry from a crumpled bin and assembles boxes.

What they do

π0 is a generalist vision-language-action (VLA) model built on the 3B-parameter PaliGemma VLM that outputs continuous actions with flow matching instead of discrete action tokens. One model is pre-trained on about 10,000 hours of the authors’ dexterous data (7 robot configurations, 68 tasks) plus OXE, Bridge v2 and DROID. It is then prompted directly or fine-tuned on curated data for long tasks such as laundry folding and box assembly.

How it works

Action expert. A separate 300M-parameter set of transformer weights handles robot state and actions and talks to the VLM only through self-attention. It predicts a 50-step action chunk with 10 flow-matching steps, reusing cached observation keys and values; on-board inference takes 73 ms on an RTX 4090.
Cross-embodiment data. Single-arm, bimanual and mobile robots share one 18-D state/action vector, zero-padded for smaller robots. Each task-robot combination is weighted by n^0.43 to down-weight over-represented data.
Pre-train, then post-train. Diverse, lower-quality pre-training data teaches recovery from mistakes; 5 to 100+ hours of curated task data teach fluent execution. A high-level VLM can split “bus the table” into language subcommands.

Results

Without fine-tuning, π0 is near-perfect on shirt folding and easy bussing and beats OpenVLA and Octo, trained on the same mixture, on all five tasks. A version trained for 160k instead of 700k steps still beats every baseline.
π0 follows language commands significantly better than π0-small (470M parameters, no VLM initialization), which carries over to higher task progress with human or VLM step commands.
Fine-tuned on new tasks, π0 generally beats OpenVLA, Octo, ACT and Diffusion Policy; pre-training improves on π0 trained from scratch by up to 2x.
On multi-stage tasks lasting 5 to 20 minutes (laundry, dryer unloading, table bussing, box building, egg packing), the full recipe scores best on every task; the authors could not solve these tasks with other methods.

Limitations

The authors pooled all available data without knowing which data helps or how to weight it. Not all tasks work reliably, and how much data near-perfect performance needs is unclear. The comparison with π0-small mixes VLM initialization with model size, and transfer to distant domains such as driving or legged locomotion is untested.

Figures

From Figure 1: A pre-trained vision-language model and a cross-embodiment dataset feed a separate action expert that ... — From Figure 1: A pre-trained vision-language model and a cross-embodiment dataset feed a separate action expert that produces continuous actions by flow matching. The model runs tasks directly from a prompt, or is fine-tuned on high-quality data for multi-stage tasks such as folding laundry or assembling a box.

From Figure 3: The pre-training mixture of the authors' own dexterous data and open-source data trains the flow-match... — From Figure 3: The pre-training mixture of the authors' own dexterous data and open-source data trains the flow-matching VLA: a PaliGemma-initialized VLM backbone plus a smaller action expert for robot states and actions, which controls several embodiments with different action spaces.

From Figure 7: Out-of-box scores on five tasks for π0 (700k steps), a compute-parity π0 (160k steps), π0-small, OpenV... — From Figure 7: Out-of-box scores on five tasks for π0 (700k steps), a compute-parity π0 (160k steps), π0-small, OpenVLA, UR5e-only OpenVLA and Octo. Even the parity version beats all baselines, and full π0 is best by a large margin.

From Figure 11: Task progress versus hours of fine-tuning data on five new tasks. π0 learns some easier tasks from sm... — From Figure 11: Task progress versus hours of fine-tuning data on five new tasks. π0 learns some easier tasks from small amounts of data, and pre-trained π0 often improves substantially over π0 trained from scratch.

Abstract

Robot learning holds tremendous promise to unlock the full potential of flexible, general, and dexterous robot systems, as well as to address some of the deepest questions in artificial intelligence. However, bringing robot learning to the level of generality required for effective real-world systems faces major obstacles in terms of data, generalization, and robustness. In this paper, we discuss how generalist robot policies (i.e., robot foundation models) can address these challenges, and how we can design effective generalist robot policies for complex and highly dexterous tasks. We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We then discuss how this model can be trained on a large and diverse dataset from multiple dexterous robot platforms, including single-arm robots, dual-arm robots, and mobile manipulators. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people and from a high-level VLM policy, and its ability to acquire new skills via fine-tuning. Our results cover a wide variety of tasks, such as laundry folding, table cleaning, and assembling boxes.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong