RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild

Tan Sang Nguyen, Yi Ru Wang, Shailesh Shailesh, Gang Yang, Jiamo Hu, Jianuo Jiang, Haoquan Fang, Zhongzheng Ren, Zhongyi Luo, Ian Chong, Aaron Tung, Santiago Pravisani, Minghui Lin, Yao Mo, Ishneet Singh, Benjamin Tee, Harold Soh, Siddhartha Srinivasa, Dieter Fox, David Hsu, Haoming Song, Maxime Ellerbach, Thomas Wolf, Pepijn Kooijmans, Ruijie He, Michael Cho, Jiafei Duan

November 2026

TL;DR A distributed real-robot benchmark runs blinded pairwise trials of five VLA policies and three frontier AI agents on four robot types at 124 sites, plus a fine-tuning track with 100 demonstrations per task. Over 5,699 comparisons, frontier agents rival VLAs, but no system exceeds 13% task success in any manipulation category.

What they do

Real-world evaluations of generalist robot policies are hard to reproduce and usually rest on a few rollouts. RoboColosseum is a distributed benchmark in which full-time operators run standardized robots at real sites (offices, libraries, parks, homes, food courts, factories) and policies are served through shared infrastructure. Over 41 days, the first study covers four embodiments (SO-101, Franka in the DROID setup, Bimanual-YAM, Unitree G1) and eight models: five VLAs (GR00T N1.7, π0.5, MolmoAct2, G0.5, LingBot-VLA v2) and three frontier AI agents.

How it works

Open Track. An operator defines a task with local objects; two hidden models run back-to-back on the same setup, and the operator records a preference, progress and a 1-10 difficulty rating.
Ranking. A Difficulty-Calibrated Rasch model jointly estimates policy ability and task difficulty, using the human rating as a prior. Active matchmaking prioritizes the model pairs whose comparison most reduces ranking uncertainty.
Fine-tune Track. 20 tasks (five per embodiment) with 100 teleoperated demonstrations each, collected at the evaluation site. Each VLA is fine-tuned with its official recipe and run for 25 trials per task, scored on success and on per-task progress criteria.

Results

Open Track: Claude Opus 5.5 rates highest on SO-101 and Bimanual-YAM, GPT-6 Astra on DROID/Franka. In 901 agent-vs-VLA pairs, agents make more progress in 394 and VLAs in 265. Success falls from 30.7% on tasks rated 1-4 to 0.4% on tasks rated 9-10.
Fine-tune Track: π0.5 reaches 84% success on DROID/Franka and 93.6% on Bimanual-YAM; no policy completes any G1 task. Rankings agree weakly across embodiments (Kendall’s W = 0.24).
Open Track rankings poorly predict fine-tuned ones (Spearman ρ of -0.20, -0.40 and 0.20 on the three shared robots).
Active matchmaking reaches about 90% agreement with the full-data ranking after 400-500 evaluations; uniform sampling needs about 1,000.

Limitations

This is an initial snapshot: model coverage differs across embodiments, and the Fine-tune Track has only five tasks per embodiment. Each policy uses its own fine-tuning recipe, so the comparison does not isolate architecture. SO-101 rankings depend strongly on which sites are included (59% pairwise agreement across site splits).

Figures

From Figure 1: Distributed sites share robot interfaces and policy serving and feed two tracks: the Fine-tune Track (... — From Figure 1: Distributed sites share robot interfaces and policy serving and feed two tracks: the Fine-tune Track (the same 100 demonstrations per task, common success and progress criteria) and the Open Track (blinded pairwise comparisons under matched local conditions).

From Figure 2: The four platforms, from left to right: Bimanual-YAM, SO-101, Franka in the DROID configuration and Un... — From Figure 2: The four platforms, from left to right: Bimanual-YAM, SO-101, Franka in the DROID configuration and Unitree G1, with cameras, grippers and computing equipment annotated.

From Figure 3: Open Track DCR ratings (Elo-style scale) on each embodiment, with 95% credible intervals; n is the num... — From Figure 3: Open Track DCR ratings (Elo-style scale) on each embodiment, with 95% credible intervals; n is the number of evaluations involving each system.

From Figure 6: Open Track executions grouped by human-rated task difficulty: (a) success rate and (b) mean normalized... — From Figure 6: Open Track executions grouped by human-rated task difficulty: (a) success rate and (b) mean normalized progress of unsuccessful executions.

Abstract

General-purpose robot policies must work across robots, tasks, and environments, yet the variation that makes physical trials informative also makes them difficult to reproduce. We introduce RoboColosseum, a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments. Our initial study evaluates five vision-language-action (VLA) policies and three frontier AI agents across four embodiments (SO-101, Franka, Bimanual-YAM, and G1 Humanoid). The Open Track compares models without task-specific adaptation in blinded, pairwise trials under matched local conditions. The Fine-tune Track measures how well VLA policies adapt to 20 tasks across all four embodiments when each policy receives the same 100 demonstrations per task. Across 5,699 pairwise Open Track comparisons at 124 sites, frontier agents rival specialized VLA policies, while all systems struggle to complete open-world tasks reliably. After fine-tuning, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM. Policy rankings show weak agreement across embodiment-specific task suites (Kendall W = 0.24). Open Track rankings poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks. Measuring partial progress reveals differences that task success alone misses. On the G1 Humanoid, no policy completes any of the five tasks, yet average progress ranges from 13.3% to 23.2% across policies. Open Track rankings stabilize as trials accumulate, and active matchmaking achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling. These results show how distributed physical evaluation can reveal where robot policies generalize, how well they adapt to new tasks, and how far they progress on tasks they fail to complete.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong