RACE: Calibrated Cumulative Scoring for Rollout-Free Robot Policy Checkpoint Selection

Changyeon Kim, Yekyung Nah, Kimin Lee

November 2026

Put Two Cubes on a real Franka: measured success rate and RACE score across training checkpoints.

TL;DR RACE ranks robot-policy checkpoints without rollouts, scoring their action predictions on held-out successful trajectories against state-dependent tolerances and compounding the scores over time. On eight simulated tasks its rank correlation with success is 0.83 versus 0.52 for validation loss, and in 1,800 real-robot trials its rankings stay aligned with success for two policy types.

What they do

Choosing which training checkpoint to deploy usually means real-robot trials for every candidate. The cheap alternative, validation loss on held-out demonstrations, often ranks checkpoints in the wrong order. RACE is a rollout-free score that ranks checkpoints from their action predictions on a fixed set of successful reference trajectories, either demonstrations or earlier successful rollouts.

How it works

Grouped distances. Actions are restored to physical units and split into groups such as translation and rotation. A chunk’s deviation per group is the mean of its largest 50% of per-step distances, with geodesic angle for rotations.
State-conditional thresholds. The k nearest reference states (frozen visual features plus proprioception) give a spread of action differences; its q-quantile is the tolerance. Deviations within it score one, larger ones decay exponentially, and a soft minimum combines the groups.
Cumulative scoring. Timestep scores are multiplied along each trajectory, so a large early deviation discounts every later prediction. The mean of this curve, averaged over trajectories, is the checkpoint score.

Results

Simulation (8 DexMimicGen tasks, 12 checkpoints x 10 seeds, 200 rollouts per checkpoint): Spearman ρ 0.83 on held-out demos, versus 0.60 for off-manifold norm, 0.52 for validation loss and -0.22 for validation MSE. With rollout or noised-rollout references RACE stays at 0.79 and 0.78, while validation loss drops to -0.10 and -0.47.
Real robot (Franka in the DROID setup, 1,800 trials): for a DROID diffusion policy, validation loss and MSE reverse the true ranking, while RACE follows success on all three tasks. For π0.5-DROID the baselines are moderately correlated and RACE leads, most clearly on the longer tasks.
Perturbing a policy only at bottlenecks costs 40 points of success and 52.7% of its RACE score; MSE changes by about 2.5%, the same as under a uniform perturbation that costs 4 points.
Removing the threshold lowers ρ from 0.76 to 0.31; averaging instead of cumulative scoring lowers it to 0.63.

Limitations

Variation among nearby successful actions only stands in for tolerable error: RACE cannot tell whether a policy would recover, or how it behaves in states its own actions reach. It gives a relative ranking, not a success-rate estimate, so selected checkpoints still need deployment trials. Fully shared hyperparameters lower ρ from 0.80 (task-tuned) to 0.69.

Videos

PnP 3 Fruit on a real Franka: measured success rate and RACE score across training checkpoints.

Figures

From Figure 1: Policy A deviates once from the demonstrated route and breaks the cup; policy B deviates more often bu... — From Figure 1: Policy A deviates once from the demonstrated route and breaks the cup; policy B deviates more often but reaches the goal. Averaged prediction error dilutes the single large error and selects A. RACE gives deviations within a reference-derived threshold a score of one, compounds lower scores over time, and selects B.

From Figure 2: RACE scoring. (a) Predicted and reference action chunks are compared within each action group. (b) Dis... — From Figure 2: RACE scoring. (a) Predicted and reference action chunks are compared within each action group. (b) Distances between action chunks at nearby reference states give a distribution whose quantile q is the state-conditional threshold. (c) Threshold-normalized deviations become group scores, which a soft minimum combines into a timestep score. (d) Running products of the timestep scores are averaged over time and across trajectories to rank checkpoints.

From Figure 4: PnP 3 Fruit, one of the three real-robot tasks for the fine-tuned pi0.5-DROID policy: the robot picks ... — From Figure 4: PnP 3 Fruit, one of the three real-robot tasks for the fine-tuned pi0.5-DROID policy: the robot picks up three toy fruits and places them on a plate.

Abstract

Assessing the deployment performance of learned robot policies during training is challenging, as directly measuring task success requires costly real-world trials for each candidate checkpoint. Averaged validation loss offers a rollout-free alternative, but its checkpoint rankings can be inconsistent with deployment performance. We analyze two limitations of this criterion: it penalizes action deviations without considering whether the policy can recover from them, and gives later reference predictions equal weight even when earlier deviations may lead execution away from the reference. We propose RACE, a novel rollout-free metric for ranking checkpoints using a fixed set of held-out successful trajectories. RACE calibrates action deviations against state-conditional thresholds derived from action differences at similar states in these trajectories. We aggregate these scores through a running product, so that a substantial deviation reduces the contribution of subsequent reference predictions. Averaging the resulting cumulative scores over time and across trajectories yields the checkpoint score. In simulation, RACE outperforms baselines across validation sources, while several baselines degrade sharply when the validation set changes. On real robots, its rankings align with deployment success across the evaluated policy backbones and tasks of varying difficulty. Perturbation analyses further show that RACE distinguishes deviations with markedly different effects on task success, even when baselines respond similarly.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong