VLA Adaptation with Real and Simulated Data: A Case Study on Robotic Chess

Matěj Kripner, Milan Straka

November 2026

TL;DR A study of fine-tuning pi0.5 to move a chess king between named squares with an SO-101 arm, using teleoperated and scripted simulated demonstrations. Pretraining on simulated demonstrations lifts real success with 128 demos from 2/25 to 18/25, and a short learning-rate cool-down adds up to 31 points in simulation.

What they do

The paper asks how much data and training a VLA needs for a new task, how to pick a checkpoint, and whether simulation can replace real demonstrations. The test task: an SO-101 arm picks up a chess king and places it on another square, with both squares given only by name (“Move the king from e4 to a5”). A MuJoCo twin of the setup lets every model run 1,000 graded trials.

How it works

Policy. pi0.5 with the vision encoder frozen. It sees an overhead and a wrist camera, six joint positions, the instruction and the current stage, and outputs chunks of 50 joint targets at 15 Hz. The language model also predicts the stage as one token.
Scripted expert. Instead of a motion planner, the simulated expert reaches with a perception error, observes after a latency and corrects, with per-episode tempo, tolerances and grasp parameters. It succeeds in 93% of episodes.
Training grid. Nested sets of 100 to 10k simulated demos and budgets from 1,000 to 237k updates; each checkpoint gets a branch that cools the learning rate to zero.
Real arm. 32, 128 or 256 teleoperated demos, starting from base pi0.5 or from the simulation-trained policy.

Results

Simulation success grows with data and updates, reaching 64% with 10k demos at 237k updates; below about 4k updates no model succeeds.
With 10k demos at 165k updates, the cooled-down model reaches 60% and the constant-rate checkpoint 49%. Validation loss is lowest where success is zero.
Real arm, 25 trials per model: with 128 demos the sim-initialized policy succeeds 18 times and the base-initialized one 2; with 32 demos, 7 against 0; with 256 demos, 14-17 against 5-10.
Negative result: filtered BC, RECAP-style advantage conditioning, coached corrections and residual RL on the policy’s own rollouts do not improve over behavior cloning beyond evaluation noise.

Limitations

One task, a single training run per setting, and only 25 real trials per model, with the real results partial. The expert’s similarity to human demonstrations is not rigorously measured, and the RECAP variants omit the original’s advantage-conditioned pretraining and human corrections.

Figures

From Figure 1: The five stages of a move on the real arm (top) and in simulation (bottom), labeled with the stage tok... — From Figure 1: The five stages of a move on the real arm (top) and in simulation (bottom), labeled with the stage tokens the policy predicts.

From Figure 3: Inputs and outputs of the policy at one frame of a real trial. The policy observes the two rectified c... — From Figure 3: Inputs and outputs of the policy at one frame of a real trial. The policy observes the two rectified camera images, the joint positions, the instruction and the current stage. It outputs a chunk of 50 joint targets and predicts whether to transition to the next stage.

From Figure 4: Success rate of the cooled-down models over updates, one curve per dataset size, with Wilson 95% inter... — From Figure 4: Success rate of the cooled-down models over updates, one curve per dataset size, with Wilson 95% intervals. Dashed lines mark the end of each size’s first epoch.

From Figure 9: Partial results in the real setting, 25 trials per model. Left: policies initialized from the simulati... — From Figure 9: Partial results in the real setting, 25 trials per model. Left: policies initialized from the simulation policy. Right: policies initialized from the base pi0.5 model. Top: success rate (%) over demonstrations and updates, where a dash marks a model without a full evaluation. Bottom: share of trials that reach each milestone, per evaluated model.

Abstract

Vision-language-action (VLA) models are trained as general-purpose foundation models for robot learning, and require fine-tuning for each downstream task and robot. This raises practical questions such as how much data and training updates are needed, how checkpoints should be selected, and whether simulation can be leveraged to alleviate the cost of real data collection. We study these questions on a task that requires both precise manipulation and spatial understanding of the scene. In this task, an SO-101 arm moves a chess king between two squares that are given only by their names, such as e4 and a5. We fine-tune $π_{0.5}$ with various dataset sizes and update budgets, using teleoperated demonstrations as well as simulated demonstrations from a randomized scripted expert that imitates the variability of human demonstrations. In simulation, the success rate scales with both data and updates. Importantly, we find that a short learning-rate cool-down adds up to 31 points and is necessary for comparing checkpoints, and that validation loss and action error are unreliable for model selection. On the real arm, we fine-tune on up to 256 teleoperated demonstrations and evaluate ten policies in 25 trials each. Partial results indicate that pretraining on the simulated demonstrations helps substantially, most clearly when real data is scarce. We also present a negative result, in which training on the policy's own rollouts with filtered behavior cloning, RECAP-style advantage conditioning, corrections from an automatic coach, or residual reinforcement learning did not yield a measurable improvement over plain behavior cloning in our setting. We release the simulator, the expert and the real demonstrations.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong