Streaming Failure Forecasting: VLA Failure Monitors That Keep Learning After Deployment

Amit Jadhav, Shaurya Beriwala, David Q. Liu, Yanfei Liu

November 2026

TL;DR A failure monitor for VLA policies that refits only its last linear layer after every episode from the success/failure outcome. It ranks failures better than the same monitor kept frozen on 4 of 5 simulated drift suites and on 434 recorded real SO-101 runs of four VLAs, at 2.18 ms per policy call on a Jetson CPU core.

What they do

Failure monitors for VLA policies are trained offline and then frozen, but after deployment the lighting, objects, camera, task and policy version change without notice. Streaming Failure Forecasting (SFF) keeps the monitor learning from the one label each episode provides, whether it succeeded, and is tested with all settings locked before each test result was opened.

How it works

Fixed features. A small two-hidden-layer network is trained on nominal warm-up episodes to predict failure at every policy call from the VLA’s internal activations (projected to 128 dimensions) plus elapsed time. Its hidden layers then stay fixed.
Online readout. After each episode, only the last linear layer is refit in closed form by discounted least squares (one 130x130 solve), anchored to the offline head so it starts out ranking exactly like the offline monitor. No episodes are stored.
Success-only threshold. The alarm threshold follows adaptive conformal inference but moves only after successful episodes, the only ones where an alarm can be false.
Evaluation. SmolVLA on LIBERO suites under injected drifts (light, object colour, camera, wear), and 434 recorded ArmnetBench runs of SmolVLA, pi0.5, pi0 and MolmoAct 2 on real SO-101 arms, replayed across unannounced task or policy switches.

Results

Simulation: within-task AUROC beats the frozen monitor on 4 of 5 test suites (+0.021 to +0.050; LIBERO-10: 0.736 against 0.707, p=0.026) and beats detect-and-reset on all five (+0.024 to +0.082).
Real runs: SFF ranks failures above the frozen monitor for every VLA. Pooled, it leads detect-and-reset by +0.113 and a full reset at the true switch by +0.044 (exploratory).
False-alarm rate on successes is 0.082-0.089 against a 0.1 target, but 0.19-0.37 under compound drift.
Cost: median 2.18 ms per policy call on one Jetson Orin Nano CPU core, with 1.0 MB of state.

Limitations

The real-robot evidence replays recorded runs; SFF has not run live. Simulation uses one policy family, gains are modest (0.02-0.05), and there is no clear difference with the authors’ own fine-tuned policy. No monitor wins on every suite: an unanchored warm-up-start readout leads on three. When nothing changes, SFF ranks slightly below the frozen monitor. It assumes a correct outcome label after every episode.

Figures

From Figure 1: A failure monitor that keeps learning after deployment. (a) On recorded runs of four VLAs on real SO-1... — From Figure 1: A failure monitor that keeps learning after deployment. (a) On recorded runs of four VLAs on real SO-101 arms, the task or the policy version switches without warning; the frozen monitor, fit on the old task, stays near chance, while SFF learns from each outcome (within-task AUROC, mean of the VLAs, exploratory). (b) A failing and a successful run after a switch. (c) Only the readout learns, from each run’s outcome.

From Figure 3: Where SFF was tested. (a) A LIBERO-Goal scene under each simulated condition, with the failure rate of... — From Figure 3: Where SFF was tested. (a) A LIBERO-Goal scene under each simulated condition, with the failure rate of SmolVLA’s recorded episodes. (b) A simulated and a real stream. Monitors are never told where a block starts. (c) The real ArmnetBench tasks and which VLA was fine-tuned for which.

From Figure 12: SFF is above the frozen monitor for every VLA. Real robot runs per VLA: SFF, the frozen monitor and d... — From Figure 12: SFF is above the frozen monitor for every VLA. Real robot runs per VLA: SFF, the frozen monitor and detect-and-reset after an unannounced switch, with 95% intervals. Detect-and-reset’s test never fired for three VLAs, so it equals the frozen monitor there.

From Figure 11: No monitor is best on every suite. Every reported monitor per simulation suite, with the SFF minus fr... — From Figure 11: No monitor is best on every suite. Every reported monitor per simulation suite, with the SFF minus frozen difference specified in advance, its 95% interval and Holm p-value.

Abstract

Failure monitors for vision-language-action (VLA) policies are trained offline and then frozen. After deployment, lighting, objects and cameras drift, and the policy itself is updated or moved to new tasks. On recorded real-robot runs, after a switch of task or policy version, a frozen monitor fit on the old task ranks failures little better than chance. We present Streaming Failure Forecasting (SFF), a monitor that keeps learning from the one label every episode provides: whether it succeeded. SFF keeps the offline monitor's network fixed and refits only its last linear layer after each episode, at 2.18 ms per policy call on a Jetson CPU core. Its alarm threshold moves only after successes, which in simulation kept false alarms near target except when two drifts came at once. With every setting locked before testing, SFF ranks failures better than the same monitor kept frozen on 4 of 5 simulated test suites under unannounced drift, and above resetting on detected drift on every suite. Carried with its locked learner settings to 434 recorded runs of four VLAs on real SO-101 arms, replayed across unannounced switches of task or policy version, it ranks failures better than the frozen monitor for every VLA and, in an exploratory comparison, above a full reset at the true switch. A deployed failure monitor need not stay frozen: refitting its readout from outcomes is cheap and helps after the changes we tested.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong