← Paper Digest

Training Reasoning Models in Simulation to Natively Use Real Robots

Jiaming Tang, Zhuoyang Zhang, Junxian Guo, Fangzhou Zhao, Yilong Zhao, Zekai Li, Xingyang Li, Yuyang Zhao, Song Bian, Zhijian Liu, Song Han

November 2026

TL;DR An open-weight 27B reasoning model learns to drive robot arms the way computer-use agents drive a PC (look at camera images, reason, output the next end-effector pose), trained with RL only in simulation and without demonstrations. On a real dual-arm robot it never saw in training it succeeds in 18 of 30 trials; the base model succeeds in none.

What they do

Robot-use agents let a vision-language model operate a robot directly, but so far only proprietary frontier models do this well. The authors train an open-weight reasoning model, Qwen3.8-27B, to use robots with reinforcement learning alone, entirely in simulation and without demonstrations, then deploy it zero-shot on a real dual-arm YAM.

How it works

  • Interface. Each round the model sees camera frames and the arm state as JSON, reasons in a chain of thought, and outputs the change in each end-effector pose. Inverse kinematics and the robot’s own controller execute it. There are no perception tools, planners or skill libraries; only the prompt’s robot description changes between robots.
  • RL. A binary task-success reward with GRPO-style group-normalized advantages, no PPO clipping, and fully asynchronous rollouts corrected by score centering. The PEAR infrastructure builds on veRL and LMCache; training took about 50,000 H100-equivalent GPU-hours.
  • Curriculum. The base model succeeds in about 2% of LIBERO episodes, so training starts on three LIBERO suites, adds LIBERO-Long, then two dual-arm RoboDojo tasks.

Results

  • LIBERO success rises from 1.6% to 74.4%, against 76.2% and 80.6% for GPT-6.1 Sol and GPT-6 Astra at maximum effort.
  • Given a different task in a training scene (LIBERO-Pro), pi0.5 drops to 0.75% while the model keeps 57.4%; pi0.5 still leads on reworded instructions and restyled objects.
  • Zero-shot on RoboLab-120, 37.3% versus 28.0% for the pi0.5 leaderboard entry.
  • Real YAM: 18/30 trials for the final model, 7/30 for the LIBERO-only checkpoint, 0/30 for the base model. No measurable loss on nine reasoning, coding and vision benchmarks.

Limitations

It is slow: a round of reasoning takes about 4.3 s on a GB300 and occasionally over half a minute. Training covers two simulators and few tasks, harder tasks need the embodiment described in detail in the prompt, and the model outputs one pose per round rather than action chunks. On the full RoboDojo benchmark, 68.5% of episodes fail by exhausting the 256k-token context.

Figures

From Figure 1: Robot-use reinforcement learning. Like a computer-use agent, the model observes the scene, reasons in ...
From Figure 1: Robot-use reinforcement learning. Like a computer-use agent, the model observes the scene, reasons in a chain of thought and outputs the next end-effector pose (top). It is trained with RL entirely in simulation, here Qwen3.8-27B in LIBERO and RoboDojo (left), and deployed on a real robot (right). Bottom: results on LIBERO, instruction following, robustness, transfer and general capabilities.
From Figure 2: One round of robot use. A computer-use agent (top) observes a screenshot, reasons and outputs a click ...
From Figure 2: One round of robot use. A computer-use agent (top) observes a screenshot, reasons and outputs a click or keystroke. The robot-use model (bottom) observes the camera views and the arm state, reasons in a chain of thought and outputs the end-effector's next pose, which inverse kinematics and the arm controller execute. Each round is added to the context, the same way in simulation and on the real robot.
From Figure 7: Real-robot evaluation on a dual-arm YAM, a robot not used in training, with no real-robot data. (a) Ge...
From Figure 7: Real-robot evaluation on a dual-arm YAM, a robot not used in training, with no real-robot data. (a) General pickup: the final model picks up objects the base model does not; bars give successes out of 20. (b) Align blocks: rounds of one trial, final versus base model; bars give successes out of 10. (c) Reasoning tasks: drawing a line version of a painting and choosing a reagent that shows whether a solution is alkaline.
From Figure 4: Comparison with pi0.5, both trained only on LIBERO (pi0.5 by supervised fine-tuning on demonstrations,...
From Figure 4: Comparison with pi0.5, both trained only on LIBERO (pi0.5 by supervised fine-tuning on demonstrations, this model by RL). (a) Success on LIBERO and the perturbed LIBERO-Plus and LIBERO-Pro. (b, c) Two LIBERO-Pro perturbations, an unseen task in a training scene and swapped object positions; arrows give the drop from LIBERO.
Abstract

Robot-use agents (RUAs) have emerged as a promising approach to general-purpose robot control. However, strong performance has so far been achieved only by frontier proprietary models such as GPT-6 Astra and Claude Fable 5.2, whereas open-weight models remain substantially less capable. We show that an open-weight 27B reasoning model can be trained to use real robots natively through reinforcement learning, entirely in simulation and without demonstrations. Like a computer-use agent (CUA), the model observes the environment, reasons in a chain of thought, and generates the next end-effector pose, without any harness. We build a Post-training Engine for Agentic Robotics (PEAR) and use it to train the model on simulated robot tasks at scale with robot-use reinforcement learning (RURL), mirroring how frontier models acquire computer use. We find that the skills acquired in simulation generalize well beyond the training tasks: they transfer to an unseen robot, to the real world, and to novel instructions. After training on LIBERO and RoboDojo in simulation, the model succeeds in 60% of our trials on a real dual-arm robot, whereas the base model succeeds in none. On LIBERO, training raises the success rate from 1.6% to 74.4%, approaching GPT-6 Astra and GPT-6.1 Sol at maximum reasoning effort. Compared with $π_{0.5}$, a vision-language-action model fine-tuned on demonstrations, our model is substantially more robust to changes in the task itself. On LIBERO-Pro, $π_{0.5}$ succeeds in only 0.75% of episodes when assigned a new task in a familiar scene whereas our model succeeds in 57.4%. Finally, the model retains its general capabilities, with no measurable degradation on nine reasoning, coding and vision benchmarks. These results highlight robot-use reinforcement learning as a promising component of foundation-model post-training.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.