Distilling Instructions into Weights: Closed-Form Edits Fix a Robot Policy's Misreading

Ali-Adeeb Abbas, Esmaeil Seraj

November 2026

Cube then banana in the bowl: the unedited policy (top) against the policy after the weight edit (bottom).

TL;DR A zero-shot pi0.5 policy misreads "put the cube then the banana in the bowl" and drops the cube early. Copying the internal effect of a clearer rewording into the VLM's MLP weights in closed form, with no gradients, raises held-out simulation success from 27/128 to 106/128.

What they do

Generalist policies often fail at one step of a task whose skills they have. This paper asks whether such a failure can be repaired by editing weights directly, with no gradients, data or prompt change. Their case: the DROID pi0.5 checkpoint, zero-shot in RoboLab (Isaac Sim), grasps the cube and lets go before the bowl, yet succeeds when the instruction is reworded explicitly.

How it works

Diagnose. Replay 21 recorded premature releases with images, state and noise fixed, changing only the wording. Moving “in the bowl” next to the cube removes most releases, pointing to a language binding error.
Distil. At each VLM layer 0-16, take 72 keys (40 prompt and state positions, 32 image cells) under the original wording and the residual difference under the explicit wording, then apply a regularized minimum-norm update to the MLP down-projection. Rank is at most 72 per layer.
Evaluate. Edited and unedited policies run from identical simulator seeds; edits are frozen on practice seeds and scored on held-out seeds with exact McNemar tests.

Results

Held-out success rises from 27/128 to 106/128 (81 fixed, 2 broken). A random-direction update of the same size scores 9/64.
A variant storing 0.16% of the parameter count scores 102/128 against 19/128, but loses head to head with the full edit (41/64 against 55/64).
The edit also lifts eyeglasses pickup (22/64 to 44/64) and cube left of bowl (8/64 to 36/64). Placed in the action expert or action head it helps less or not at all.
Edits aimed at perception, missing competence, contact physics or the gripper threshold did not help. Rerunning unchanged weights flips about a third of paired outcomes.

Limitations

One zero-shot checkpoint, simulation only, 16 to 128 pairs per comparison. The method needs a known-good wording as input, and the released tool has run on one task. The edit is low-rank but dense, changing 13.4% of stored values, and transfer was not predicted in advance.

Videos

Transfer to picking up the eyeglasses, unedited against edited policy.

A held-out run with the original wording: unedited model against the edited model on the same run.

Figures

From Figure 1: One held-out start, unedited and edited. Top: wrist-camera inputs at four matched control steps. Botto... — From Figure 1: One held-out start, unedited and edited. Top: wrist-camera inputs at four matched control steps. Bottom: commanded gripper (filled where closed) and logged events. Right: held-out success over 128 starts. With the instruction unchanged, the edit removes the release three steps after the grasp.

From Figure 2: Bars give the share of 21 recorded premature-release decisions that still release when replayed offlin... — From Figure 2: Bars give the share of 21 recorded premature-release decisions that still release when replayed offline with images, robot state, and noise fixed. (a) Rewordings. (b) Weight edits in the order they were built, then variants of the distilled edit.

From Figure 3: Each bar splits a comparison’s paired starts into the four outcome types, with the number of pairs and... — From Figure 3: Each bar splits a comparison’s paired starts into the four outcome types, with the number of pairs and of fixed and broke pairs at right. On its own task the distilled edit flips pairs almost only one way. Failed edits and unchanged-weight reruns flip many pairs both ways.

Abstract

Generalist robot policies often fail at one step of a task they can otherwise perform. We ask whether such a failure can be repaired by editing the policy's weights directly, without gradients, training data, or a change to the instruction. Told to put the cube then the banana in the bowl, a zero-shot $π_{0.5}$ vision-language-action policy in RoboLab releases the cube before it reaches the bowl. Replaying 21 of these decisions under rewordings shows that it binds "in the bowl" to the wrong object. Closed-form instruction distillation copies the per-layer effect of an explicit rewording into the language model's MLP weights. On 128 held-out seeds, success rises from 27 to 106 episodes, with 81 fixed and 2 broken ($p=7\times10^{-22}$), while an update of the same size in random directions leaves success unchanged. The edit also lifts two other tasks without detectable harm on the remaining 13, and a variant that stores 0.16% of the parameter count keeps most of its effect. Placed in the action expert or action head instead, the same distillation helps less or not at all. Our released tool reproduced the fix end to end on new held-out seeds, from 8/64 to 36/64. Edits aimed at perception, a missing grasp, contact physics, or the gripper threshold did not help, and of seven blind Claude Opus editors, only the one given the misread instruction produced an edit that passed held-out scoring. These results suggest that weight edits work best when a known-good input already exists to distil. Because reruns with unchanged weights flip about a third of paired outcomes, evaluations should pair seeds, rerun the unedited policy, and confirm gains on held-out seeds.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong