Reins: Frontier Multimodal Models Driving Real Robot Arms on Long-Horizon Tasks, Zero-Shot

Pengfei Ye, Zhiyang Dou, Yuxiang Ma, Haonan Chen, Heng Yang, Edward H. Adelson

November 2026

TL;DR Reins lets an off-the-shelf frontier model drive a dual-arm robot through tool calls. The model chains atomic actions, each with a check that stops the chain at its first failure, while a second instance of the model watches the cameras and reports what changed. With natural-language prompts and no task-specific code, GPT-6.1 Sol succeeds in 88% of real episodes across five long-horizon tasks.

What they do

Frontier models can plan and call tools, but on real robots they struggle with long-horizon, contact-rich tasks, partly because they are blind while the arm moves and while they think. Reins is a harness in which an off-the-shelf model operates two 6-DoF AirBot Play arms through perception tools and atomic actions, with no skill library, task code or robot training. It is tested with GPT-6.1 Sol, GPT-6 Astra and Claude Opus 5.5 on five tasks: a shell game, opening and filling a drawer, folding a towel, folding a shirt, and sweeping with a brush and dustpan.

How it works

Tools. The model has eighteen tools. find (SAM 3 on RGB-D) and measure turn an object name or pixel into a 3D point, accurate to about 1.6 mm with the wrist camera. move_until_contact stops when joint current indicates contact, and close_gripper reports whether anything is held.
Guarded chains. do runs up to eight tool calls, each with a check on its result and an optional repeat-until condition, and stops at the first failed check. Model latency is 41-78% of episode time, so each decision covers 1.6-3.4 checked steps on average.
Narrator. A second instance of the model watches the head camera at 1 frame/s, and its account of what changed comes back with every tool result.
Twins. A separate tool fits parametric object models to SAM masks to build a MuJoCo scene from one head-camera frame. It was not used in the episodes.

Results

Over five episodes per task, GPT-6.1 Sol succeeds in 88% of real episodes. Without the harness it fails every shell game and takes 8 times as long. The three models perform alike overall, and all three open the drawer every time.
The shell game, in which the mugs are shuffled while the model waits, separates them. Without the narrator, Sol wins 2 of 5. Sol’s narrator answers in 1-6 s and sees a shuffle in 8-11 windows, while Astra’s and Opus’s take 3-40 s and see it in 2-4. Astra’s narrator still tracks the red object in 4 of 5 episodes, Opus’s in none.
In the twins, 11 of 14 movable objects pass the simulation-ready checks with mask IoU 0.90-0.99. A scripted version of each task runs to completion in simulation.

Limitations

Task knowledge enters only through prompts; the drawer and sweep prompts add grip and pose hints. Contact is sensed only from joint current and gripper width. The evaluation covers one platform, with five episodes per condition and success judged by the operator. A slow narrator reports fast changes late.

Figures

From Figure 1: Left: three calls from a Sol drawer episode. Each result reports what the robot measured, a chain of p... — From Figure 1: Left: three calls from a Sol drawer episode. Each result reports what the robot measured, a chain of pulls stops when its condition fails, and the narrator's account arrives with the next result. Middle: the five tasks. Right: simulation-ready twins of the scenes, each built from one head-camera frame.

From Figure 3: Twins of the five task scenes, each built from one head-camera RGB frame. Columns, from left: the fram... — From Figure 3: Twins of the five task scenes, each built from one head-camera RGB frame. Columns, from left: the frame, the twin rendered from the same camera, the twin from another view, and the start, middle and end of a scripted task in MuJoCo.

Abstract

Frontier models such as GPT-6.1 Sol, GPT-6 Astra and Claude Opus 5.5 can now plan and call tools, yet they still struggle to drive real robots through long-horizon, contact-rich manipulation, blind while the arm moves and while they think. We present Reins, a harness in which an off-the-shelf model operates a dual-arm robot with 3D perception tools and atomic actions that report contact and grip. The model chains actions in one call, each with a condition that must hold, stopping at the first that fails; a second instance, the narrator, watches the cameras continuously and reports what changed with every result. From images, Reins also builds a simulation-ready twin of the scene for continual policy learning. With natural-language prompts and no task-specific code, GPT-6.1 Sol succeeds in 88% of real episodes on five tasks; without the harness, it fails every shell game, taking 8 times as long. Three frontier models perform alike but differ in tracking a scene that changes while the actor thinks. Code will be released upon publication.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong