CoachSI: Self-Improvement of Robot Policies with a Language Coach
TL;DR When a robot policy stalls partway through a long task, a frozen VLM names the next step, chosen from the policy's own atomic instructions. The successful coached rollouts are relabeled with the original task instruction and used to fine-tune the policy. In simulated RoboCasa365 kitchens, unassisted success on 18 long-horizon tasks rises from 47.4% to 59.3%, against 49.4% after fine-tuning on the same budget of human demonstrations.
What they do
Generalist policies run single skills reliably but stall when the skills are chained into long-horizon tasks: once stalled, 97% of continuations under the task instruction make no further progress in 320 steps. CoachSI uses a frozen, off-the-shelf VLM as a coach that tells the policy its next step, and then trains that behavior back into the policy so that it needs no coach at test time.
How it works
- Language coach. At each action chunk, a frozen Qwen3.8-27B sees the three camera views and the task’s list of next-step instructions, written in the policy’s atomic-instruction format. It returns the first unfinished step, or KEEP. It never outputs actions.
- Hindsight relabeling. Successful coached rollouts keep their observations and actions, but every coaching instruction is replaced by the original task instruction.
- Fine-tuning. The OpenWAM world action model is fine-tuned with 20% coached data and 80% human demonstrations. Only the self-attention layers of its video backbone are updated, and the action backbone stays frozen.
Results
- On 500 stalled states from OpenWAM and the Xiaomi-Robotics-1 VLA, the correct next-step instruction raises subgoal completion 4.6x (4.2x on the WAM, 6.4x on the VLA). An image-only VLM coach recovers 91% of the oracle’s gain at 320 steps.
- Unassisted success on 18 long-horizon tasks in unseen scenes (1,080 episodes): 47.4% pretrained, 49.4% with extra human demonstrations, 51.9% with self-collected successes, 59.3% with CoachSI using a state-based coach and 56.8% with an image-only coach.
- Keeping the coach in the loop at test time instead gives 41.9%. Dropping hindsight relabeling gives 51.7%.
- Per task: FillKettle rises from 18.3% to 70.0% and HeatMug from 41.7% to 86.7%.
Limitations
All results are from simulation, and the paper reports no real-robot test. The coach only picks from a given list of next-step instructions for each task. The headline 59.3% uses a coach that reads object and robot state as text, and fine-tuning is shown only for the WAM.
Videos
Figures



Abstract
Generalist robot policies execute atomic skills reliably but fail when the same skills are composed into long-horizon tasks. Self-improvement from the policy's own experience does not close this gap: experience collected by the policy alone stays within its own distribution and reproduces what it can already do. We show that what the policy lacks is not a skill but the next step. We present CoachSI, a self-improvement method in which a frozen, off-the-shelf vision-language model (VLM) coaches the policy in language: it watches the robot and names the next step in the format of the policy's atomic instructions, and never outputs an action. Successful coached rollouts are relabeled in hindsight with the original task instruction and used to fine-tune the policy. The fine-tuned policy is then deployed on its own. On states where a World Action Model and a Vision-Language-Action model have stalled, naming the next step raises subgoal completion by 4.6x over the unchanged task instruction, and the frozen VLM supplies it from the same camera images the policy sees. After fine-tuning, unassisted success on long-horizon tasks in unseen scenes rises from 47.4% to 59.3%, 9.8 points above fine-tuning on human demonstrations and 7.3 points above fine-tuning on self-collected successes at an equal data budget. With a coach that sees only camera images, it reaches 56.8%, above both baselines. The gain is obtained with the action backbone frozen: language coaching teaches the policy what should happen next, and the skills it already has do the rest.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
