Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement

Michal Korniak, Marco Bagatella, Ralf Römer, Michal Nauman, Yarden As, Pranav Vignesh, Manan Agarwal, Leander Diaz-Bone, Piotr Zalewski, Mikołaj Czarnecki, Lenart Treven, Shashwat Saxena, Sreyas Venkataraman, Sarvesh Patil, Zhiyuan Zhou, Daniel Marta, Oier Mees, Nur Muhammad Mahi Shafiullah, Carmelo Sferrazza, Bhavya Sukhija, Angela P. Schoellig, Andreas Krause, Max Simchowitz, Andrew Wagenmaker

November 2026

Simulated wine-bottle pick, an unseen object, where Best-of-N fails after 64 training tasks and succeeds after 144.

TL;DR They post-train the pi0.5 VLA online with RL on up to 144 simulated pick tasks at once and test on held-out scenes and objects. With critic-based recipes the gains grow with the number of training tasks, up to +58 points on unseen scenes and +39 on unseen objects, while filtered fine-tuning generalizes much less.

What they do

RL post-training of vision-language-action models is usually done one deployment task at a time. This paper asks whether a generalist policy can be improved online on many tasks at once, and whether that improvement carries over to tasks it never trained on. It is an empirical study of existing recipes, not a new algorithm.

How it works

Setup. The base policy is pi0.5. In the MolmoSpaces pick suite, training splits of 1, 4, 16, 64 and 144 tasks (k objects with k scenes each, base success about 30%) are evaluated on unseen scenes of the training objects and on six unseen objects. LIBERO-90 is a second benchmark.
Protocol. 10 rounds, each collecting 20 episodes per task with the policy frozen and then updating.
Three recipes. Filtered supervised fine-tuning on successes (FSFT); Best-of-N, which keeps the VLA frozen and picks among 32 sampled action chunks with a learned TD critic; and OGPO, which updates the policy toward actions the critic scores highly.

Results

On training tasks with many tasks, OGPO and BoN beat FSFT by 43-50 and 15-21 points. A single BoN run on 16 tasks needs 2.9x fewer gradient steps than 16 single-task runs for similar gains.
Single-task post-training causes forgetting on unseen objects. With more training tasks, critic-based recipes gain up to +58 points over the base policy on unseen scenes and +39 on unseen objects.
Object diversity matters more than scene diversity: scaling scenes to 16 hardly beats the base policy on unseen objects.
On LIBERO, no generalization to unseen tasks emerges with up to 64 training tasks.

Limitations

Simulation only, manipulation only, and tasks were chosen so the base policy already makes some progress. Multi-task policies still trail single-task experts slightly on their own training tasks, and the largest runs took 200 hours.

Figures

From Figure 4: Final success rate in MolmoSpaces on the training tasks (left), unseen scenes (center) and unseen obje... — From Figure 4: Final success rate in MolmoSpaces on the training tasks (left), unseen scenes (center) and unseen objects (right) as the number of training tasks grows. The critic-based BoN and OGPO generalize more than FSFT.

From Figure 6: OGPO success rate on eight unseen scenes (left) and eight unseen objects (right) when scaling the scen... — From Figure 6: OGPO success rate on eight unseen scenes (left) and eight unseen objects (right) when scaling the scenes of one object or the number of objects with one scene each. Only scaling objects surpasses the base policy on both.

From Figure 8: Each method is trained on one object, a box or a spatula, and evaluated on the unseen knife. For BoN a... — From Figure 8: Each method is trained on one object, a box or a spatula, and evaluated on the unseen knife. For BoN and OGPO, training on the similarly shaped spatula transfers better, whereas FSFT shows no such pattern.

Abstract

The introduction of generalist robot policies has enabled a single, large model to learn to perform a variety of tasks from a vast pre-training corpus. While these models often exhibit powerful zero-shot generalization capabilities, they typically require an adaptation phase to achieve sufficient performance in deployment. Iterated self-improvement via reinforcement learning (RL) has demonstrated promise in enabling this adaptation, yet such approaches have been largely explored in single-task settings - where the goal is to simply improve on a narrow deployment task - which lies in stark contrast to the multi-task capabilities of the base policy. In this work we ask whether generalist robot policies are capable of multi-task, generalist improvement in deployment - targeting many diverse tasks simultaneously - and what algorithmic recipe enables this. To answer this question, we first develop a set of structured evaluations to test generalist improvement capabilities, and systematically evaluate algorithmic and practical design choices regarding value estimation, policy extraction and resource allocation. We show that not only can generalist policies be improved online across diverse tasks, but that the right recipe can enable generalization and increase performance on tasks not considered during online training. We characterize when this generalization occurs and show that both the training task distribution and the use of value-based RL are critical. To the best of our knowledge, this is among the first evidence for online generalist improvement for robot foundation models.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong