Off-Policy RL with Online Inference Tuning: My Solution to the CoRL 2026 RoboEval Challenge

Ilia Larchenko

November 2026

TL;DR A single pi0.5-based policy that is also its own value function is improved with off-policy RL on the exact challenge score, while every rollout doubles as a Thompson-sampling trial of about 20 inference parameters. With scripted full-reward demonstrations added, it reaches 0.987 RDS and 99.25% success on the official seeds of the CoRL 2026 RoboEval bimanual challenge.

What they do

The CoRL 2026 RoboEval challenge scores eight bimanual Franka tasks in MuJoCo with the RoboEval Diagnostic Score (RDS), which combines success and stage progress with safety, efficiency and coordination measured against human demonstrations. This solo entry trains one policy for all eight tasks: a generic RL loop that optimizes the score directly, plus generated data for behaviors the demonstrations never show.

How it works

Policy as its own critic. pi0.5 gets a value query token predicting success, stages, completion and the behavior terms still to come, and a Q head that picks the best of N sampled action chunks.
RL on the exact score. The reward is the RDS linearized around the current policy; advantages drive AWR sampling and RECAP-style advantage conditioning, with classifier-free guidance at inference.
Tuning while collecting. Each of the 2000 episodes per cycle draws about 20 inference parameters (flow steps, temperature, CFG scale, best-of-N, speed) by Thompson sampling on its score.
Data beyond the demos. Scripted policies with privileged simulator state generate full-reward demonstrations, and failed rollouts are continued from branch points into recovery episodes.

Results

The generic loop alone, with only the organisers’ demonstrations and its own rollouts, reached 0.85 RDS in about two days on one GPU.
The submitted policy scores 0.990 RDS on 500 held-out episodes per task and 0.987 RDS with 99.25% success on the official seeds; five of eight tasks score exactly 1.000.
Replayed as a policy, the organisers’ demonstrations score 0.875 RDS on average; only 1 of 141 StackTwoBlocks and 2 of 99 PackBox demonstrations reach every stage.
Checkpoint averaging and per-task parameter search added about +0.01 RDS (0.981 to 0.990).

Limitations

Simulation only. Getting past the demonstrations’ ceiling took privileged scripted demonstrations (around 200k generated), more than 240k rollouts and about 600 GPU-hours, which the author calls not data-efficient. Scripts only work where the simulator is trustworthy and the task is easy to script, and after the switch to cleaner scripted data the policy visibly lost some regrasps and retries. A 50-seed run varies by about 0.005 RDS.

Figures

From Figure 1: The eight tasks, each at the start and at the end of one full-reward episode.

From Figure 3: The training loop on two machines, a trainer and a collector, which run the same commit and share data... — From Figure 3: The training loop on two machines, a trainer and a collector, which run the same commit and share data and models through one Hugging Face bucket.

From Figure 4: The policy: pi0.5 plus a value query, a Q head and advantage conditioning (token + AdaRMS) that enable... — From Figure 4: The policy: pi0.5 plus a value query, a Q head and advantage conditioning (token + AdaRMS) that enables classifier-free guidance.

From Figure 7: The RDS of every collect (2000 episodes on held-out seeds, each with its own draw of inference paramet... — From Figure 7: The RDS of every collect (2000 episodes on held-out seeds, each with its own draw of inference parameters; mean per 2k steps) over the policy’s training steps, up to step 60k, where training stopped. At step 42k training moved from one H200 to eight B300s with a larger batch (dashed line).

Abstract

The CoRL 2026 RoboEval challenge has eight bimanual tasks, and it ranks a policy not only on whether it succeeds but on how: every stage of the task, done safely, efficiently and with both arms in coordination, scored against human demonstrations. My solution has two parts. The generic part, carried over from my LeHome challenge solution, is off-policy RL on a $π_{0.5}$-based policy that is also its own value function. It is trained with AWR and RECAP on advantages of the exact official score, in a loop where every rollout is at the same time a Thompson-sampling trial of about 20 inference parameters. Out of the box, it took the policy to 0.85 RDS on held-out seeds in two days (and it was still very far from convergence). The competition-specific part adds data that local exploration cannot find: generated demonstrations that reach the full reward, and recovery episodes that start from the policy's own failures. The submitted policy reaches 0.990 RDS on held-out seeds and 0.987 on the official seeds.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong