$π_{0.5}$: a Vision-Language-Action Model with Open-World Generalization

Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, Ury Zhilinsky

★ Must read April 2025

TL;DR π0.5 extends π0 by co-training one vision-language-action model on mobile-manipulator data, other robots, subtask labels and web data, and uses it both to pick the next subtask and to output actions. Although 97.6% of its pre-training examples come from sources other than household mobile manipulation, it cleans kitchens and bedrooms in homes it never saw.

What they do

π0.5 tests how far a vision-language-action model generalizes in homes it has never seen. Building on π0, it trains on about 400 hours of mobile-manipulator data from about 100 homes, plus a much larger mixture of other sources: static arms in other homes, lab data from many robot types, subtask annotations and web data. From a high-level prompt such as “place the dishes in the sink”, the policy cleans kitchens and bedrooms in new homes.

How it works

One model, two levels. The model first outputs a text subtask (“pick up the cup”), then predicts a 50-step action chunk conditioned on it. A 2B PaliGemma backbone plus a 300M action expert.
Pre-training (280k steps). Plain next-token prediction of text, bounding boxes and FAST-tokenized actions. 97.6% of examples in this phase do not come from mobile manipulators doing household tasks.
Post-training (80k steps). Adds a flow-matching action expert, trains on successful mobile and static-arm episodes plus web and subtask data, and adds verbal instructions: experts steering the robot step by step with language commands.
Control. Target poses for arms and torso and base velocities at 50 Hz, tracked by PD controllers with no planner or collision checking.

Results

In three unseen real homes it completes multi-stage kitchen and bedroom tasks lasting about 2-5 minutes.
With data from 104 locations it matches a control trained on the test homes themselves; without the co-training pre-training, both baselines are significantly worse.
Removing other-robot data (ME or CE) significantly hurts. Removing web data mainly hurts unseen object categories and subtask inference.
It beats π0 and π0-FAST+Flow. Its own subtask inference beats a human oracle and GPT-4 as the high-level policy; verbal instructions are about 11% of the high-level examples, yet removing them significantly hurts.

Limitations

Unfamiliar drawer handles and hard-to-open cabinets, partial observability (an arm hiding a spill), and a high-level policy that gets distracted, opening and closing a drawer repeatedly. Prompts are simple and context is modest, which the authors flag for tasks spanning rooms or needing memory of where things are stored.

Figures

From Figure 2: π0.5 cleaning a kitchen in a home that was not in the training data. Given general tasks (close the ca... — From Figure 2: π0.5 cleaning a kitchen in a home that was not in the training data. Given general tasks (close the cabinets, put the items in the drawer, wipe the spill, put the dishes in the sink), it predicts subtasks such as "pick up the plate" and emits low-level actions.

From Figure 3: Two training stages. Pre-training combines data from diverse robots (actions as discrete FAST tokens),... — From Figure 3: Two training stages. Pre-training combines data from diverse robots (actions as discrete FAST tokens), high-level subtask prediction and web data. Post-training specializes the model for mobile manipulation, adds verbal instructions from human supervisors, and uses flow matching for continuous actions. At inference the model first infers a subtask, then predicts actions for it.

From Figure 4: Example training tasks. Pre-training uses mobile manipulator data (MM), non-mobile robots in diverse e... — From Figure 4: Example training tasks. Pre-training uses mobile manipulator data (MM), non-mobile robots in diverse environments (ME), laboratory cross-embodiment data (CE), high-level subtask prediction (HL) and multi-modal web data (WD). Post-training adds verbal instructions (VI) and drops CE.

From Figure 8: Average progress on four test tasks (dishes in sink, items in drawer, laundry basket, make bed) improv... — From Figure 8: Average progress on four test tasks (dishes in sink, items in drawer, laundry basket, make bed) improves with more training environments. The dashed green line and green bar are a model that includes the test homes in training; the best model reaches similar performance without seeing them.

Abstract

In order for robots to be useful, they must perform practically relevant tasks in the real world, outside of the lab. While vision-language-action (VLA) models have demonstrated impressive results for end-to-end robot control, it remains an open question how far such models can generalize in the wild. We describe $π_{0.5}$, a new model based on $π_{0}$ that uses co-training on heterogeneous tasks to enable broad generalization. $π_{0.5}$\ uses data from multiple robots, high-level semantic prediction, web data, and other sources to enable broadly generalizable real-world robotic manipulation. Our system uses a combination of co-training and hybrid multi-modal examples that combine image observations, language commands, object detections, semantic subtask prediction, and low-level actions. Our experiments show that this kind of knowledge transfer is essential for effective generalization, and we demonstrate for the first time that an end-to-end learning-enabled robotic system can perform long-horizon and dexterous manipulation skills, such as cleaning a kitchen or bedroom, in entirely new homes.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong