Latent RL Priors for Data-Efficient Dexterous Manipulation

Yixiao Wang, Peiyi Chen, Mustafa Mukadam, Shreekant Gayaka, Masayoshi Tomizuka, Son Tran

November 2026

Simulation: the arm-hand system grasps the held-out brush and aligns it with a target pose.

TL;DR Distill a large-scale simulated RL policy for a KUKA arm with a dexterous hand into a frozen flow-matching action decoder, then learn only a visual front end from a few demonstrations. In simulation it beats π0.5 and GR00T N1.7 fine-tuned on the full datasets using 10x fewer demos on door lifting and 5x fewer on bimanual assembly.

What they do

RL in simulation can learn rich dexterous control, but the resulting policies rely on privileged object and goal states. Latent RL Prior distills such a policy into a flow-matching action decoder, freezes it, and learns only a visual front end from a small set of image demonstrations. The same single-arm decoder also drives both arms in a bimanual task.

How it works

Privileged pretraining. They fine-tune the released SimToolReal teacher on procedural pose alignment (KUKA iiwa with a SHARPA hand; randomized object shape, size and density) and collect 100,000 successful rollouts. A state tokenizer and Q-Former turn privileged history into latent context tokens. A transformer decoder cross-attends to them, with proprioception and flow time entering through AdaLN.
Visual adaptation. A DINOv2-Small encoder and a new visual Q-Former map images directly to the context tokens, supervised only by the flow-matching action loss: no privileged labels or latent targets. The decoder stays frozen.
Bimanual. Each arm gets its own query group in one Q-Former, and self-attention between the groups coordinates the arms. The shared decoder then runs once per arm.
Downstream demonstrations are rollouts of per-task SAPG RL policies; everything runs in MuJoCo Warp.

Results

With limited demonstrations it beats full fine-tuning of π0.5 and GR00T N1.7 on all three tasks. On door lifting, 50 demos beat both baselines trained on the full 500; on bimanual assembly, 1,000 beat their full-data result (5x fewer).
At the relaxed 0.05 m pose threshold the gap narrows, reaching parity with π0.5 at 4,000 demos.
Latent transfer beats routing images through the pretrained state encoder and is more stable across seeds. Fine-tuning the decoder gives no benefit at any tested learning rate.
On heavier objects never seen in the demos, door lifting keeps 32.6% success at 5.0 kg, against 12.9% for π0.5.

Limitations

Everything, including demonstration collection, is in simulation, so real-robot performance is untested. The method needs a capable privileged teacher and a lot of simulated experience, and the frozen prior can inherit the teacher’s failure modes. Only one arm-hand system is tested, with no transfer to other morphologies.

Videos

Simulation: two arm-hand systems grasp two parts and insert one into the other.

Simulation: the hand lifts a door to reveal the cube behind it.

Figures

From Figure 2: Pretraining aligns varied procedural objects with target poses (left, inset shows object templates). T... — From Figure 2: Pretraining aligns varied procedural objects with target poses (left, inset shows object templates). The downstream tasks are target pose alignment with a held-out brush, door lifting and bimanual assembly (right), all reusing the same pretrained action decoder.

From Figure 1: (a) Privileged pretraining: a state tokenizer and Q-Former supply task context to an action decoder co... — From Figure 1: (a) Privileged pretraining: a state tokenizer and Q-Former supply task context to an action decoder conditioned on flow time and single-arm proprioception. (b) Visual adaptation: an image encoder and Q-Former learn the context tokens while the decoder stays frozen. For bimanual assembly, left- and right-arm queries interact in the same Q-Former, and each output group conditions the shared decoder with its own arm's proprioception.

From Figure 4: Success rate against number of demonstrations, compared with full fine-tuning of GR00T N1.7 and π0.5, ... — From Figure 4: Success rate against number of demonstrations, compared with full fine-tuning of GR00T N1.7 and π0.5, for (a) target pose alignment at the strict 0.015 m and relaxed 0.05 m thresholds, (b) door lifting and (c) bimanual assembly. Mean and standard deviation over three seeds.

From Figure 8: Success as object mass increases, for (a) target pose alignment at the relaxed 0.05 m threshold and (b... — From Figure 8: Success as object mass increases, for (a) target pose alignment at the relaxed 0.05 m threshold and (b) door lifting. The leftmost point in each plot is the mass used in the visual demonstrations.

Abstract

Learning dexterous manipulation from limited visual demonstrations is challenging because policies must acquire both perceptual grounding and precise control, while reinforcement learning (RL) in simulation can acquire rich control behaviors from large-scale interaction with privileged information. We introduce \textbf{Latent RL Prior}, a two-stage framework that reuses this control knowledge for data-efficient visual policy learning. During privileged pretraining, we distill large-scale RL rollouts into a flow-matching action decoder conditioned on proprioception and latent task context. During visual adaptation, we freeze the decoder and learn a visual interface that maps images to its latent context using action supervision alone, without privileged labels or latent targets. The same decoder can further be shared across arms by learning coordinated visual contexts for bimanual manipulation. Across three simulated tasks, our method outperforms full-parameter fine-tuning of $π_{0.5}$ and GR00T N1.7 in the low-data regime; on door lifting and bimanual assembly, it surpasses full-data baselines using $10\times$ and $5\times$ fewer demonstrations, respectively. Further analysis shows that direct visual-to-latent conditioning improves adaptation performance and training stability over routing visual predictions through the pretrained state encoder. Robustness to unseen mass variation and greater linear accessibility of contact-force information in decoder representations further provide evidence of retained RL-acquired control knowledge. Project website: https://yixiaowang7.github.io/latent-rl-prior-page/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong