Few-Shot Residual Adaptation of Pretrained Latent World Models

Jakub Suliga, Pankhuri Vanjani, Nisarga Nilavadi, Wolfram Burgard, Rudolf Lioutikov

November 2026

TL;DR Small residual adapters, trained self-supervised on three interaction episodes, correct a frozen latent world model both inside its predictor and at its output. On a real Franka pushing task, success recovers from 0% to 90% under a center-of-mass shift and from 3.3% to 100% under changed friction.

What they do

A latent world model trained under one set of physical conditions can fail when an object’s dynamics change at deployment. This workshop paper keeps a pretrained model (DUET-DINO, a V-JEPA 2-AC variant) fully frozen and learns lightweight residual corrections from its own prediction errors, using three interaction episodes and no rewards or knowledge of what changed. The task is pushing a box to a goal pose with a closed gripper, with a hidden weight shifting the center of mass or a cloth changing friction under half the track.

How it works

Planning. A one-step cross-entropy-method controller samples actions, predicts side- and wrist-camera latents, and executes the action whose prediction is closest to the encoded goal image. The controller is never changed.
Inside the predictor. Adapter+ modules are inserted at two points in each of the 24 Transformer blocks of both predictor heads (96 modules).
Outside the predictor. A shared token-wise residual MLP corrects the predicted latent. Both residuals are zero-initialized, and together add 29.2M parameters, 4.53% of the frozen predictor.
Adaptation. After each of the first three episodes, the internal adapters are trained for 25 steps on next-latent prediction, then frozen while the external MLP is trained for 25 steps.

Results

The frozen model reaches 85.3% success in simulation and 86.7% on the real robot under normal dynamics, then 0% under the center-of-mass shift and 3.3% under modified friction.
Real robot (30 runs each): the dual-site adapter reaches 90% under the center-of-mass shift and 100% under modified friction; full finetuning gets 20% and 100%, LoRA 30% and 66.7%.
Simulation, center-of-mass shift: dual-site 56.7%, against 32.7% for the best inside-only method (LoRA) and 24.0% for the best output-only correction; training the two sites sequentially beats joint training (44.0%).
An adapter trained on unshifted dynamics removes most of the same prediction error but reaches only 6.0% success under the shift, against 72.0% for the shift-specific adapter.

Limitations

Only center-of-mass and friction shifts on one pushing task, with a separate adapter per shift. The adapted model drops to 54.0% on the original dynamics, and adapters transfer poorly between simulation and the real robot. Each real control step takes about 28 s.

Figures

From Figure 1: Dual-site latent correction in a frozen world model. The frozen predictor rolls out sampled actions, c... — From Figure 1: Dual-site latent correction in a frozen world model. The frozen predictor rolls out sampled actions, corrected inside every block and once more on its output. The CEM controller ranks the corrected predictions by latent distance to the encoded goal and executes the best action. The next observation supplies the target of the self-supervised loss that trains both residuals.

From Figure 2: The pushing task in simulation and on the real robot, each shown at the start of an episode and after ... — From Figure 2: The pushing task in simulation and on the real robot, each shown at the start of an episode and after a successful push.

Abstract

Pretrained latent world models can support visual planning and control, but their predictions may fail when physical dynamics change at deployment. We study whether such models can adapt from only a few interactions without modifying their pretrained weights or identifying the changed physical parameters. We introduce a few-shot residual adaptation approach that learns self-supervised corrections from latent prediction errors. The corrections are applied both inside the latent predictor and directly to its predicted latent state, while the pretrained world model remains frozen. We evaluate the approach in simulation and on a real robot under center-of-mass and friction shifts. These changes reduce the success of the frozen model to near zero. Using three interaction episodes for adaptation, our dual-site approach recovers 90\% success under the center-of-mass shift and 100\% under the modified friction on the real robot. Ablations show that correcting both locations outperforms single-site adaptation. Our diagnostics further suggest that much of the learned correction reduces a general prediction mismatch, while successful control also depends on shift-specific information captured during adaptation.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong