Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing

Michal P. Podolinsky, Yunhao Yang, Daniel Little, Hong Lu, Rohan Siva, Olin Goudey, Neel P. Bhatt, Matthias Scheutz, Ufuk Topcu

November 2026

TL;DR VLA-Tug is a small LSTM probe on a VLA's backbone embeddings that predicts failures before they happen and names the likely failure mode, with an alarm threshold calibrated to bound both false alarms and missed early detections. On held-out LIBERO-X skills it raises early-detection area by 30% over the strongest learned baseline, at 0.12 ms per step.

What they do

VLA-Tug is a runtime failure monitor for vision-language-action policies. A small probe reads the VLA’s own backbone embeddings at every policy step and outputs a failure score and a predicted failure mode. A calibrated threshold on the score raises the alarm, ideally before the failure happens, so an expert model can step in. They test it on π0.5 and GR00T-N1.7 over 228 simulated tasks (LIBERO, LIBERO-Occ, LIBERO-X) and 8 real tasks on a Kinova Gen3 arm.

How it works

Probe. Per step, the mean backbone tokens of the agent camera, wrist camera and prompt are concatenated into a 6144-d vector. A 256-unit LSTM feeds three MLP heads: failure score, a 10-way failure mode, and an auxiliary CLIP-aligned description of the current action.
Pre-onset supervision. Failed rollouts are labeled positive only before the failure time and cut off there. Gemini labels failure time, mode and a timeline from each rollout video, for about one cent per failed episode.
Calibration. Learn-then-Test picks a threshold that jointly guarantees false-positive rate ≤ α and early true-positive rate ≥ β. This bounds the chance that an alarm is a real failure, and the method abstains when the targets can’t be met.

Results

Early-detection area on unseen tasks: best in four of six settings. Against SAFE-LSTM it scores -1% on new-task, +14% on new-signature and +30% on new-skill splits. On the real robot the lead is small (25.8 vs 25.1), and it does not lead on LIBERO-10 with GR00T.
Calibration on LIBERO-X: both targets held in every certified run on seen tasks. On unseen tasks the false-alarm target held in 40-86% of runs, versus at most 21% for max-conformal.
Failure mode: top-1 balanced accuracy up to 149% (seen) and 79% (unseen) above a majority baseline on LIBERO-X.
Trains in 6.8 s and adds 0.12 ms per step on a GPU.

Limitations

The guarantees assume exchangeable episodes and hold only empirically on unseen tasks. The real-robot set is too small (12-14 calibration successes per run) to certify the calibration targets. The steering demos use simple heuristics and are not evaluated.

Figures

From Figure 1: (a) A recurrent encoder maps the VLA backbone embedding into a latent failure space; a score head with... — From Figure 1: (a) A recurrent encoder maps the VLA backbone embedding into a latent failure space; a score head with a calibrated trigger raises the alarm and a mode head predicts the failure mode. (b) VLA-Tug alarms before the failure, allowing early stopping or intervention by an expert model.

From Figure 3: A successful (top) and a failed (bottom) rollout with their timeline, failure time and failure mode la... — From Figure 3: A successful (top) and a failed (bottom) rollout with their timeline, failure time and failure mode labels.

From Figure 5: Early detection on seen (top) and unseen (bottom) tasks: the share of failures detected at or before t... — From Figure 5: Early detection on seen (top) and unseen (bottom) tasks: the share of failures detected at or before the failure time against the false-positive rate, for VLA-Tug and the learned and action-based baselines. Panel titles give the benchmark, the policy and the held-out split.

From Figure 6: Real-robot steering on "put the mushroom on the plate". Without steering the gripper misses the mushroom. With steering, the probe predicts a grasp failure, the VLA is paused while the wrist camera is moved 0.05 m toward the mushroom, and the resumed policy succeeds.

Abstract

Vision-Language-Action models (VLAs) enable general-purpose embodied intelligence, but their performance is brittle under distribution shift, where unanticipated failures can damage equipment or endanger people. Deploying VLAs in real-world systems therefore requires a runtime monitor that is timely, calibrated, and informative while being efficient enough to run on the edge. Existing methods fail to fulfill these requirements. We present VLA-Tug, a lightweight probe over the latent layers of a VLA backbone that improves early failure detection and bounds the missed-detection and false-alarm rates, with minimal inference overhead. We show that supervising on the pre-failure segment of a trajectory increases early failure detection at matched false-positive rates by up to 30% over the strongest learned baseline. Under shift to unseen tasks, the bounds on missed-detection and false-alarm rates remain valid in up to 86% of runs, versus at most 21% for baselines. Unlike existing methods, VLA-Tug predicts the failure mode to enable corrective actions, raising top-3 accuracy on unseen tasks by up to 44% over a majority baseline. VLA-Tug enables real-time monitoring where the well-timed and informative probe supports lightweight steering that improves task success. We release a dataset of 3,180 annotated manipulation episodes to support future work on latent failure detection in VLAs at https://vla-tug.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong