PyRUA-DataFarm: LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data

Kewei Zhang, Juncheng Ma, Guodong Lv, Duomin Wang, Feixiang Ruan, Ruiyang Si, Chuanrui Zhang, Yuxuan Chen, Hao Chen, Yichen Liu, Yintianrun Zhang, Yufan Deng, Enze Xie, Daquan Zhou

November 2026

Towel folding: simulator rollout (left) and its photographic rendering (right), head and wrist views.

TL;DR A coding agent rebuilds a real task in simulation from one calibration recording, and a video model conditioned on real frames renders the simulated rollouts as photographic multi-view video with exact action labels. On five real dual-arm tasks, policies trained only on this data succeed in 93 of 100 trials against 84 for teleoperated data, and 87 against 46 under an unseen tablecloth and coloured light.

What they do

Teleoperated demonstrations cost robot and operator time for every episode, and every new task starts from scratch. DataFarm gives the teleoperator’s job to a coding agent. From a data request and one short calibration recording of the robot in the target scene, the agent rebuilds the task in simulation. The simulator supplies exact actions, and a video model conditioned on real frames renders each rollout as photographic multi-view video that a policy trains on like teleoperated data.

How it works

Grounding. The agent identifies the robot and objects, fits cameras and metric scale, and labels each physical quantity as measured, fitted, assumed or unknown. Quantities a video cannot reveal, such as mass and friction, are randomized rather than guessed.
Expert against a hidden judge. A separate role turns the task contract into a judge that is frozen before the expert controller is written. The author sees only verdicts on development seeds, and the expert is admitted on held-out initial states.
Rendering. Edge maps of the simulator video drive a ControlNet that fixes arm pose, objects and timing, while real frames supply appearance. Head and both wrist views are generated as one video so they agree; a new tablecloth or light is a text change.
Admission. Automatic checks (missing or extra objects, event timing, stability) and an independent reviewer agent compare every render with its simulated state.

Results

Five bimanual tasks on a dual-arm PiPER, π0.5 post-trained on 100 episodes per source, 20 trials per task: farm data 93/100 vs teleoperation 84/100 in the original scene, and 87 vs 46 under an unseen tablecloth and coloured light.
Rendering the same trajectories with simulator domain randomization instead gives only 26 and 21 successes.
Adapting a task costs $21 of API usage on average plus $5 for a render contract, then $0.80 per episode. One hour of three-view data costs $107, against a $117 teleoperation quote in China.

Limitations

The simulator must be able to reproduce each task’s skills, and qualification probes exist only for cloth. Dynamics the calibration episode does not show are randomized, not recovered, which may not be enough for tasks that depend on precise dynamics. Automatic video checks can miss defects. The evaluation covers one dual-arm robot and five tasks.

Videos

Block-in-drawer episode rendered under warm lamp light, with the simulator view inset.

Nine stack-cups episodes with different cup placements.

Figures

From Figure 1: (a) The same frame of a simulator rollout and of its rendering for each of the five tasks; the renderi... — From Figure 1: (a) The same frame of a simulator rollout and of its rendering for each of the five tasks; the rendering is photographic and keeps the simulated arm pose, objects and contacts. (b) From a request and one calibration episode, the agent builds the task in simulation and the video model renders each rollout in the original scene or in a new one described by text. (c) Real-robot successes of policies trained only on farm data or on as many teleoperated episodes. (d) Mean costs over the five tasks.

From Figure 2: Overview of DataFarm on the pull-drawer task. (1) A request and one calibration episode are the only i... — From Figure 2: Overview of DataFarm on the pull-drawer task. (1) A request and one calibration episode are the only inputs. (2) A coding agent rebuilds the task in simulation and writes an expert against a judge it cannot see; the simulator runs the expert from many initial states. (3) A video model conditioned on real frames renders every rollout in the scene's appearance. (4) An independent reviewer compares each render with its simulated state before it enters the dataset. The cost plot is schematic.

From Figure 4: Real-robot results. (a) Successes of π0.5 policies trained on 100 episodes from each source, in 20 tri... — From Figure 4: Real-robot results. (a) Successes of π0.5 policies trained on 100 episodes from each source, in 20 trials per task, in the original scene (left) and under an unseen tablecloth and coloured light (right). (b) Unseen-scene trials in which the policy trained on teleoperated data fails (top) and the policy trained only on farm data succeeds (bottom).

Abstract

Teleoperation remains the default way to collect robot demonstrations, but every episode costs robot and operator time, and every new task starts from scratch. We ask whether a coding agent can take over the teleoperator's job. In our DataFarm, the agent receives a data request and one calibration episode of the robot in the target scene and rebuilds the task in simulation: it writes the scene, an expert controller, and a success judge. The simulator supplies exact actions, and a video model conditioned on real images renders each rollout as photographic video, generating all camera views together consistently. The output is real-ready data: synchronized camera views labelled with the robot's actions, on which a policy can be trained directly and deployed on the real robot. Independent judges certify the expert and every rendered episode before admission. On five real-robot tasks, policies trained only on farm data match policies trained on as many teleoperated episodes (93 vs. 84 successes in 100 trials), and under an unseen tablecloth and coloured light they keep 87 successes where the teleoperation-trained policies fall to 46. Adapting a task is fully automatic and costs \$21 of API usage on average, paid once per task, plus \$5 for a render contract; every further episode costs \$0.80 on average for simulation, rendering, and review, with no robot or operator. One hour of three-view data for a task costs \$107 on average, these costs included, slightly below a quote of \$117 for teleoperation in China and below \$290 in labour alone in North America.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong