TwinBridge: Digital Twin-Driven Policy and Reward Transfer for Real-World Reinforcement Learning in Laboratory Robotics

Shu'ang Sun, Yibo Qiu, Haoliang Ye, Zan Huang, Ao Tian, Kan Ding, Ronald Xu, Mingzhai Sun

November 2026

TL;DR TwinBridge builds Gaussian-splatting digital twins of lab benches, pretrains a policy in them, and distills simulator-derived progress labels into a visual reward model for human-in-the-loop RL on the real robot. Across five lab tasks it reaches 98% mean autonomous success with at most 25 minutes of real-world interaction per task.

What they do

Real-world RL can learn precise lab manipulation, but it needs demonstrations of every stage and rewards that show intermediate progress. TwinBridge gets both from a digital twin of the bench: a policy pretrained in simulation, and a visual reward model trained on simulator labels, which together drive human-in-the-loop RL on the real robot.

How it works

Twin. Isaac Sim physics with 3D Gaussian splatting for rendering. Rigid-body poses move the Gaussians, so the rendered images follow the robot-object interaction.
Physics-grounded rewards. Each task has ordered events with prerequisites (moving a tube only counts after a secure grasp). A ledger credits each event once, plus any new record in geometric progress, and potential-based shaping turns this credit into a dense reward.
Visual distillation. A GRU on frozen Octo features predicts events and progress from images. It is trained on simulator-labeled sequences plus a small set of real sequences annotated with events.
Real-world RL. HIL-ConRFT actor-critic, initialized from the simulation policy, with operator interventions. Hardware: a JAKA Minicobo with a parallel gripper, and a LinkerHand L7 holding a pipette.

Results

Final autonomous success is 100% on four tasks and 90% on tube loading (98% mean). Without simulation initialization it is 92%; sparse-reward HIL-ConRFT reaches 44%, HG-DAgger 76% and HIL-SERL 4%.
Simulation initialization reaches 80% rolling success sooner: about 3.9 vs 9.2 min on tube pickup and 7.6 vs 12.9 min on lid removal.
On held-out real trajectories the reward model scores 99.30% event F1 and 0.889 Spearman correlation between predicted progress and time.
In simulation, dense rewards reach 100%, 90% and 90% rolling success on pickup, lid removal and loading; sparse rewards reach at most 50%.

Limitations

The paper has no limitations section. Each task needs its own twin, hand-specified events and prerequisites, and some annotated real sequences. Real-world learning relies on a human operator, and the initial gripper position is randomized only within ±20 mm horizontally.

Figures

From Figure 1: A digital twin supplies two things for real-world online RL: a policy pretrained on simulated demonstr... — From Figure 1: A digital twin supplies two things for real-world online RL: a policy pretrained on simulated demonstrations and RL, and a visual process-reward model. On the real robot, the frozen reward model guides policy refinement with human interventions. The bottom row shows the five laboratory tasks.

From Figure 2: Reward construction and transfer. (a) In simulation, ordered events and new progress records get one-t... — From Figure 2: Reward construction and transfer. (a) In simulation, ordered events and new progress records get one-time credit in an execution-history ledger, which defines a shaping potential. (b) Labeled simulated sequences and a few annotated real ones train a causal visual predictor. (c) The frozen predictor updates the same ledger from real images to give online rewards.

From Figure 3: The five tasks at three stages each, with every real camera image paired with its digital-twin render.

From Figure 5: Real-world online learning. Ten-episode rolling success (top) and intervention rate (bottom) over real... — From Figure 5: Real-world online learning. Ten-episode rolling success (top) and intervention rate (bottom) over real interaction time for TwinBridge with and without the simulation policy, HIL-ConRFT, HG-DAgger and HIL-SERL.

Abstract

Online reinforcement learning offers an effective approach to precise laboratory manipulation, yet costly real-world demonstrations and limited access to dense process rewards hinder its practical application. We present TwinBridge, a digital twin-driven framework that supplies policy initialization and dense visual rewards for real-world online learning. We construct digital twins by combining 3D Gaussian splatting with physics simulation. Here, simulator states provide task events and geometric progress under physical prerequisites and ordering constraints, yielding rewards for policy pretraining with simulated demonstrations and reinforcement learning. We distill this supervision into a visual reward model. During real-world interaction, its predictions are combined with execution history to compute dense rewards for human-in-the-loop policy refinement. Across five real-world laboratory tasks, TwinBridge achieves a final mean autonomous success rate of 98% with at most 25 minutes of real-world interaction per task. Experiments show that dense rewards improve interaction efficiency over sparse rewards, and simulation-based policy initialization further accelerates learning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong