JustRUA: A Controlled Study of GPT-6 Robot-Use Agents

Zefan Wang, Yufan Dang, Xuanwei Shi, Mingqi Gao, Cheng Yin, SikYuen Tam, Hanyu Liu, Yasheng Wang, Yaxi Lu, Songyang Zhang, Xin Cong, Yuan Yao

November 2026

insert_tubes in simulation: the arms pick up tubes and set them into the rack, head and wrist cameras.

TL;DR A controlled study of the harness that lets GPT-6 Astra drive a bimanual arm directly, testing the loop, action interface, reused experience and scene geometry one at a time. Keeping full history, short motions, predicted depth and coordinate-solving code raises the RoboDojo simulation score from 29.7 to 50.3 with no learned policy.

What they do

GPT-6 Astra can drive a bimanual arm with no robot training: a harness turns each of its decisions into end-effector targets for a motion planner. Systems built this way keep raising scores on the RoboDojo simulation benchmark, but each changes several things at once. This study tests the common designs one at a time on a single testbed, along four axes: the loop around the model, the action interface, experience reused from other episodes, and scene geometry in each observation. JustRUA is what survives.

How it works

Testbed. The official L3 harness run inside a Codex session at medium reasoning effort: three RGB views and arm state in, world-frame gripper targets out, CuRobo plans the motion. Each design is compared with the configuration kept so far.
Evaluation set. 111 episodes covering all 42 tasks instead of the official 2,100, with four fixed layouts for the 23 tasks whose scores vary most. A separate model reviews every first-round episode to count empty grasps, drops and collisions, and the cost of each episode is reported.
What JustRUA keeps. The full episode history (the official harness keeps images of only the last two observations); motions cut after 20 control steps (about 0.8 s) so the model looks again; compact gripper readings; depth predicted by Depth Pro from RGB alone; and code the model writes to solve 3D coordinates from pixels, with no cap on decisions.

Results

The overall score rises from 29.7 with the official harness to 50.3, and success from 22.5% to 43.3%. Full history plus the 20-step cap alone reach 38.0.
Relative targets (33.9), named skills (34.6) and model-chosen motion lengths (37.9) do not beat plain absolute targets (38.0).
Predicted depth cuts first-round empty grasps from 41 to 16 and scores 42.4, against 46.0 with simulator depth, although its distances are off by tens of centimetres. Grasp affordances drawn onto the images did not beat depth alone.
Demonstrations and a self-written skill library change behaviour but do not clearly beat the default (35.2-39.9 vs 38.0). An episode costs at least $4.94 on the default testbed and $12.53 with JustRUA.

Limitations

Simulation only, with one benchmark, one model and one run per configuration. Designs are tested along a single path, not in every combination, and the evaluation set was chosen from the authors’ own runs, so its scores are not leaderboard scores. Most remaining failures are specific to one task: plans the arm cannot execute, unplanned knocks and slipping grasps.

Videos

fold_clothes: both grippers close on the shirt cuffs and lift them.

pick_from_conveyor_by_image: the gripper closes on the rim of the cup shown on the card.

Figures

From Figure 1: From the official L3 harness to JustRUA. Left, every design tested, by axis, with its score on the 111... — From Figure 1: From the official L3 harness to JustRUA. Left, every design tested, by axis, with its score on the 111-episode evaluation set. Right, the designs JustRUA keeps, in the order they were added; the dashed line is the cost of one episode.

From Figure 2: The testbed loop and the four axes of the study. The model issues a command, the unchanged planner tur... — From Figure 2: The testbed loop and the four axes of the study. The model issues a command, the unchanged planner turns it into a motion, and the simulator returns the next observation and, at the end, the score. Blue outlines mark what JustRUA changes from the official L3 harness.

From Figure 7: (a) A head-camera frame with the raw Depth Pro prediction the model receives, the same prediction scal... — From Figure 7: (a) A head-camera frame with the raw Depth Pro prediction the model receives, the same prediction scaled to the robot geometry, and simulator depth. (b) Behaviour counts with robot readings only, with simulator depth and with predicted depth.

From Figure 9: Coordinate solving in insert_tubes. At its first decision the model casts rays from pixels of the left... — From Figure 9: Coordinate solving in insert_tubes. At its first decision the model casts rays from pixels of the left tube through the head camera onto the table plane; at decision 15 it triangulates three holes of the rack and the tube tip from two cameras. Bottom, the code of the first run and its output.

Abstract

GPT-6 Astra can drive a robot arm directly through a fixed harness, and scores of such robot-use agents on RoboDojo have risen quickly since. Reported systems differ in harness and test subset and often change several designs at once, so it is hard to tell which choices produce the gains. We take these designs apart and test them in one framework, along the harness around the model, the action interface, the reuse of experience and the scene geometry given to the model. A lean harness that keeps the full episode history and stops each motion early goes a long way. Reusing experience changes how the model acts but is not a free lunch, and scene geometry helps most, even when depth is predicted from RGB alone. JustRUA starts from the official L3 harness and keeps only what matters most. Even with no learned policy, self-improvement loop or task-specific tuning, this simple harness raises the score on the RoboDojo simulation benchmark substantially, from 29.7 to 50.3.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong