← Paper Digest

RoboBuilder: Agentic Robot Co-Design with Real-world Video Feedback in the Loop

Ghadi Nehme, Dahyun Nam, Akshay Govind Srinivasan, Faez Ahmed

November 2026

TL;DR Four coding agents design brick-part robots and their motor programs to cross ten obstacles, and every design that passes in simulation is built and tested for real (135 robots). 66% of simulated passes cross the real course, and redesigning from videos of real failures raises levels 1-2 from 73% to 93%.

What they do

RoboBuilder is a benchmark in which coding agents design both a robot and its open-loop motor program from a real inventory of brick-compatible parts, motors and one custom 3D-printed part. Every design that passes in simulation is built by hand and run on a physical copy of its course, and each real failure goes back to the agent as video.

How it works

  • Tasks. Ten obstacles, such as a wall, a rope and a seesaw, at three levels. Agents must cross within 30 simulations, then get 30 more plus the printed part, then make the design lighter or faster, and finally build one robot for all ten obstacles.
  • Learned simulator. A MuJoCo model built from public part data passed 15 designs that all failed for real. Opus 5.5 requested bench tests and fitted the simulator to filmed runs, adding joints that pull apart, slipping gears and per-tyre grip. It predicts 35 of 38 labelled runs.
  • Hardware loop. Shown its failure video, the agent may make a small simulator change, accepted only if the old design now fails as filmed, then redesigns. The redesign must pass 3 of 5 perturbed simulations before it is built.

Results

  • Agents (Opus 5.5, Sonnet 5.5, GPT-6 Astra, GPT-6.1 Sol) pass 98% of level-1 tasks, 70% at level 2 and 48% at level 3. Random search and a genetic algorithm pass none.
  • A single robot for all obstacles crosses at most 60% of them in simulation and 20-60% for real.
  • 66% of simulated passes cross the real course (77%, 68% and 42% by level). One round of the loop cuts level-1 and level-2 failures from 18 to 5, taking real success from 73% to 93%.

Limitations

The simulator is fitted from one fixed camera and bench tests without force data, so some parameters cannot be identified. Each loop correction is fitted to a single failure, and redesigns are retested on the same course, so the loop measures recovery rather than transfer. Robots run open-loop, Light and Fast designs were not built, and each agent ran once per task.

Figures

From Figure 2: Twelve of the 135 agent-designed robots, built by hand on their real courses: whegs, printed hoops and...
From Figure 2: Twelve of the 135 agent-designed robots, built by hand on their real courses: whegs, printed hoops and cam wheels, cord trolleys, lever arms, legs and a spoked wheel that vaults the wall.
From Figure 3: The benchmark. Agents combine inventory parts, motors and one custom CadQuery part printed in up to fo...
From Figure 3: The benchmark. Agents combine inventory parts, motors and one custom CadQuery part printed in up to four copies, under design rules, and write an open-loop program; below are the tasks with their simulation budgets and the ten obstacles, each at three levels.
From Figure 3: One turn of the hardware loop (level-2 bumps, Opus 5.5). Spoked wheels cross in simulation but spin ag...
From Figure 3: One turn of the hardware loop (level-2 bumps, Opus 5.5). Spoked wheels cross in simulation but spin against a ridge for real; after the agent lowers the friction of plastic on the bump rods the simulator fails the same way, and the redesign on worm-driven whegs crosses for real.
From Figure 7: One loop case per agent: (a) the original design failing for real, with its failure mode; (b) the rede...
From Figure 7: One loop case per agent: (a) the original design failing for real, with its failure mode; (b) the redesign crossing in simulation and (c) on the real course.
Abstract

Can AI agents design robots that work in the real world? We introduce RoboBuilder, a benchmark in which coding agents co-design a robot's hardware and program from standard parts, motors and a custom 3D-printed part, and every design that passes in simulation is built and tested on a physical course. Simulation alone is not enough: under public part data, none of 15 initial passing designs crossed the real course, so an agent learns the simulator from measurements it requests and videos of real runs. Across four frontier coding agents, ten obstacles at three difficulty levels, 16,427 simulations of 12,774 distinct robots and 135 physical robots built in an estimated 240 hours, agents solve nearly all easy obstacles and invent mechanisms our search baselines do not reach, but falter at harder levels, when one robot must cross every obstacle, and in the real world, where 66% of simulated passes hold across all levels. Closing the loop helps: shown a video of its real failure, an agent redesigns the robot, sometimes correcting its simulator, and removes 72% of level-1 and level-2 failures. RoboBuilder measures progress toward agents that engineer real machines. Project Page: https://ghadinehme.com/robobuilder.github.io/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.