Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Embodied First Exam

Kangrui Wang, Qineng Wang, Changhe Chen, Pingyue Zhang, Chi Gui, Feishi Wang, Canyu Chen, Wenli Xiao, Xiaoyu Huang, Tonghe Zhang, Linjie Li, Zihan Wang, Minyeong Kim, Yiyou Sun, Mingkai Deng, Jinyu Hou, Qi Zhu, Han Liu, Eric P. Xing, Huaxiu Yao, Zhiwen Fan, Yuejiang Liu, Dawn Song, Pieter Abbeel, Li Fei-Fei, Jiajun Wu, Yuke Zhu, Guanya Shi, Ruohan Zhang, Manling Li

November 2026

TL;DR A simulation benchmark of 1,156 robot tasks on 47 robots in which general-purpose coding agents choose how to control the robot (direct actions, written controllers or a trained policy). On 456 hard core tasks the best agent solves 34% when it only gets sensor observations and, by default, no resets.

What they do

Coding agents are starting to be used to control robots, but robotics benchmarks are built for visuomotor policies. The Embodied First Exam connects unmodified coding agents (a model in a harness such as Codex or Claude Code) to simulated robots and lets each agent decide how to act: send actions directly, write a program that controls the robot, or train a policy. It has 1,156 tasks in 35 suites on 47 robots; 9 suites are new and 26 are adapted from existing benchmarks.

How it works

  • Containers and a verifier. Each task starts from a fixed initial state with a time limit and compute caps. A verifier outside the agent’s container grades the outcome.
  • Three settings per task. Privileged: full simulator state and unlimited resets, meant as an approximate upper bound. Standard: only proprioception and sensor data such as RGB-D through an API, the simulation pauses while the agent thinks, and no resets unless the task grants a budget. Realtime: like Standard, but the world keeps running.
  • Core set. Results are on 456 hard tasks, left after removing tasks judged easy or solved by a screening agent.
  • Failure labels. 427 core tasks are split into stages, and a separate coding agent labels the stage where each failed attempt went wrong.

Results

  • Eight agents, one attempt per test. The strongest, GPT-6 Astra in Codex, solves 77% of Privileged tests but 30% of the Standard tests of the same 391 tasks. The top three agents solve 24.1% to 33.6% of the 456 Standard tests.
  • In controlled runs, adding resets lifts GPT-6 Astra from 28% to 53%, exact object state to 45%, both to 68%.
  • Stages needing precision suffer most outside the Privileged Setting: the failure rate of assembling rises from 34% to 80%.
  • Changing only the harness moves GPT-6 Astra from 34% (Codex) to 22% (Claude Code).

Limitations

One attempt per test, so ability and run-to-run variation cannot be separated. Stage and control-approach labels come from agents, and both the screening and labeling models share a family with the top agent. 392 tasks have no reference solution. Everything is in simulation except a one-task buzz-wire demonstration with a physical arm.

Figures

From Figure 1: Examples of the control approaches, each with an excerpt of the agent’s commands or code. Dark tabs ma...
From Figure 1: Examples of the control approaches, each with an excerpt of the agent’s commands or code. Dark tabs mark the approaches used (model is direct control), and the framed line is a policy.
From Figure 2: The evaluation protocol. In the Privileged Setting the agent runs in the same container as the simulat...
From Figure 2: The evaluation protocol. In the Privileged Setting the agent runs in the same container as the simulator, resets as often as it wants and submits a trajectory that the verifier replays. In the Standard and Realtime settings the agent sees only the robot API, and the verifier grades the live episode.
From Figure 4: The tasks of the exam: the 9 suites built for the exam, each with a frame and its number of tasks, and...
From Figure 4: The tasks of the exam: the 9 suites built for the exam, each with a frame and its number of tasks, and a frame from each of the 26 adapted suites, ordered by their main robot embodiment, from single arms to aerial robots.
From Figure 7: Success rate of each agent on the core set, with solved tests in parentheses.
From Figure 7: Success rate of each agent on the core set, with solved tests in parentheses.
Abstract

Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots. Existing robotics benchmarks are built around visuomotor policy learning, leaving it open whether coding agents can reliably control diverse robots under physical-world constraints. We therefore introduce the Embodied First Exam, a unified, comprehensive, and agent-native evaluation framework where coding agents can choose their own control approach: direct control, code to control, or train a policy to control. We hope it serves as a first exam for coding agents before they enter the physical world, with 1,156 tasks in 35 suites on 47 robots, including 9 new suites that go beyond pick-and-place to spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. Each task can be run in up to three evaluation settings. The Standard Setting mimics the real world: the agent receives only sensor observations and, by default, cannot reset the environment. Its Realtime variant keeps the environment running while the agent reasons. The Privileged Setting gives the agent access to the full simulator state and unlimited resets, allowing it to search for an oracle solution for training purposes. Success rates in this setting serve as an approximate upper bound. On the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, achieves a success rate of 34% in the Standard Setting, leaving much room for improvement. To see where agents fail, we label the stage at which failed attempts go wrong and compare failures across agents, settings, and stage categories.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.