← Paper Digest

Genie Sim Benchmark: Real-to-Sim Fidelity and Evaluation

Di Yang, Da Huang, Yang Lv, Fan Yang, Yi Lu, Zheyuan Xing, Linting Guo, Jia'en Shen, Wenjun Sun, Maoqing Yao, Qian Wang

November 2026

TL;DR Pair simulated manipulation tasks with matching real ones to test whether the Genie Sim Benchmark predicts real-robot performance. Across 22 robot-task pairs on four robot configurations, fine-tuned pi0.5 policies score nearly the same in simulation and on hardware (Pearson r = 0.9686, MAE 0.04), backed by robot and object system identification.

What they do

A simulation benchmark is only useful if its scores say something about real robots. This report tests Genie Sim Benchmark, which has 78 simulated manipulation tasks covering instruction following, spatial reasoning, robustness and general manipulation. Simulated tasks are paired with real ones, and the task-level comparison is backed by robot and object system identification.

How it works

  • Paired evaluation. Each simulated layout mirrors a real test layout, with the same criteria and trial counts. Task-specific pi0.5 policies fine-tuned on real-robot data run in both. People score the real runs and the simulator scores its own, as a stage-averaged Score. The 22 pairs cover 16 tasks on G1 Omnipicker, G2 Swiftpicker, G2 Wuji and Arx X5, from target selection and sorting to bimanual handover, stacking and cloth folding.
  • Robot identification. For both G2 arms, friction sweeps, multi-frequency sines and a coupled Fourier trajectory fit link inertias, armature, friction and torque bias, under physical-feasibility constraints.
  • Object identification. Five assets (kettle, gum box, sponge, shuttlecock, pen) are fit to one recorded real interaction each, then replayed from recorded commands in MuJoCo, MuJoCo-Warp and PhysX.

Results

  • Across the 22 pairs, real and simulated Scores correlate at Pearson r = 0.9686 (MAE 0.0405, RMSE 0.0520), with gaps from -0.110 to +0.080.
  • On 14 G2/Arx pairs, identification raised r from 0.9518 to 0.9669 and cut MAE by 20%. Eight gaps shrank, one grew (Arx size recognition) and five were unchanged.
  • On a 32.7 s teleoperation recording left out of identification, per-joint position RMSE was 0.006-0.035 rad and torque RMSE 0.74-4.0 N m.
  • Fitting cut kettle error from about 39-41 mm to 25-26 mm in all three engines, while the gum box barely changed. Frozen MuJoCo parameters failed in PhysX (the sponge was not lifted, the shuttlecock and pen rolled away). Native PhysX contact settings then let all three complete their replays.

Limitations

The numbers are descriptive, with no confidence intervals, and the G1 scores come from earlier Genie Sim 3.0 experiments without identification. Each object study is one recorded episode, and its “held-out” windows are later parts of the same recording. The authors state that the identification work does not establish a causal reduction in task-score gaps.

Figures

From Figure 3: The Real2Sim workflow: paired task set, recorded real execution, robot and object identification, fixe...
From Figure 3: The Real2Sim workflow: paired task set, recorded real execution, robot and object identification, fixed-parameter replay, and real-sim comparison of task Scores. Robot identification, object replay and paired task evaluation are separate measurements.
From Figure 5: Closed-loop consistency of the identified G2 arm with frozen parameters. Top, position and joint-side ...
From Figure 5: Closed-loop consistency of the identified G2 arm with frozen parameters. Top, position and joint-side torque RMSE for seven joints under Fourier, friction and sine excitation. Bottom, distributions of absolute position and torque error per joint.
From Figure 6: Object observation error (mm, log scale) for the kettle, gum box and sponge replayed in MuJoCo, MuJoCo...
From Figure 6: Object observation error (mm, log scale) for the kettle, gum box and sponge replayed in MuJoCo, MuJoCo-Warp and PhysX. Hollow markers are baseline models, filled markers fitted ones; red marks an increase in error.
Abstract

We evaluate Genie Sim Benchmark through paired real-simulation tasks and complementary system-identification studies. Across 22 robot-task pairs spanning 16 task names, task-specific pi0.5 policies fine-tuned on real-robot data achieve a descriptive Score correlation of 0.9686, MAE 0.0405, RMSE 0.0520, with simulation-real differences from -0.110 to +0.080. G2 crsB experiments assess long-horizon closed-loop position and joint-side torque agreement, while five representative object cases examine identification and cross-engine contact transfer. A separate 32.6503 s teleoperation trajectory, excluded from mechanical identification and servo-gain search, evaluates external position targets through simulated motion-controller command shaping and a calibrated position servo. Across both arms, per-joint position RMSE spans 0.00614-0.03522 rad and current-derived torque RMSE spans 0.73545-4.00099 N m. Native PhysX adaptations recover three recorded interaction tasks, with case-dependent alignment gains. These results support task-specific simulation evaluation through joint assessment of task outcomes and physical-response consistency.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.