← Paper Digest

Teaching Vision-Language-Action Models to Stop Safely

Zheyuan Liu, Chuhan Wang, Jinduo Guo, Yue Huang, Xiangchi Yuan, Ningshan Ma, Meng Jiang

November 2026

Bottle placement after mid-task takeover, left to right: the original policy places it, NPO-RT drops it and SAFI keeps holding it.

TL;DR SAFI fine-tunes a VLA to stop a prohibited task even when it takes over halfway through. It does so by imitating a "freeze and keep the grip" action at every state of that task's demonstrations while still imitating the robot's other tasks. In LIBERO simulation, prohibited-task completion after takeover falls from 99.7% to 1.3% with no dropped objects, and the other tasks keep 93-97% success.

What they do

Methods that make a VLA refuse a task (unlearning, refusal fine-tuning) are usually tested from the task’s start state. The authors instead let the original OpenVLA-OFT policy begin a prohibited task, then hand control to the edited policy early or mid-task. Several edited policies that never start the task on their own still finish it after takeover, or abandon it by dropping the held object. SAFI (State-Aware Fallback Imitation) teaches the policy to hold still instead.

How it works

  • Fallback targets. Every action chunk in the prohibited-task demonstrations is replaced by a zero-motion arm command that repeats the first gripper command. From any state along the task, the robot freezes and keeps its grip.
  • Joint imitation. LoRA on the language model plus the action head is trained for 300 updates. The loss is an equally weighted L1 sum over the fallback targets and the original actions of the retained tasks.
  • Applicability check. SAFI is used only on tasks where a scripted hold from takeover states prevents completion without incidents.
  • Evaluation. Simulated LIBERO Goal, Spatial and Object tasks, plus a BadLIBERO backdoor attack. Metrics: target success after takeover, physical incidents (drops, falls), action saturation and retained-task success.

Results

  • After takeover, prohibited-task success falls from 99.7% to 1.3%, with no incidents and no saturation. Refusal FT reaches 11.6% success and 4.1% incidents. GA and GradDiff reach about 12%, but only with commands pinned at the controller limits. On BadLIBERO, SAFI has zero attack success, incidents and saturation.
  • Retained tasks keep 93.7%, 96.9% and 93.1% success on Goal, Spatial and Object (originally 94.4%, 99.2% and 94.2%).
  • The held object ends 3.1 cm from where it was at takeover, with 97.3% gripper contact. Every other edited policy leaves it 27.6 cm or more away.
  • Ablation: without fallback imitation, target success stays at 99.7-100%. Without retained imitation, retained success drops to 0%.

Limitations

Everything is in simulation, with one VLA (OpenVLA-OFT), selected LIBERO tasks and a single training seed. Holding is the only fallback, so SAFI applies only where holding is safe. Unstable objects or contact already underway would need other responses, such as setting the object down.

Videos

Stove task after mid-task takeover (original policy, NPO-RT, SAFI from left to right), where only SAFI leaves the stove off.

Figures

From Figure 2: Two gaps in task suppression on LIBERO-Goal. (a) Stove activation rate from the initial state and afte...
From Figure 2: Two gaps in task suppression on LIBERO-Goal. (a) Stove activation rate from the initial state and after early and mid-task takeover. (b) Cumulative bottle drops after mid-task takeover: some policies leave the task unfinished but drop the bottle.
From Figure 3: Overview of SAFI. Holding targets are built across prohibited-task demonstrations and learned jointly ...
From Figure 3: Overview of SAFI. Holding targets are built across prohibited-task demonstrations and learned jointly with the original actions of retained tasks. A checkpoint is chosen by suppression and retained success, then audited against baselines after mid-task takeover.
From Figure 8: Rollouts after matched mid-task takeover. The original policy completes both prohibited tasks. NPO-RT ...
From Figure 8: Rollouts after matched mid-task takeover. The original policy completes both prohibited tasks. NPO-RT drops the bottle and turns the stove on. SAFI keeps the bottle held and the stove off.
Abstract

Vision-language-action (VLA) models enable robots to perform diverse tasks from visual observations and language instructions. While prior work has explored task suppression and refusal, stopping prohibited behavior once execution is underway and controlling the resulting physical response remain challenging. Our evaluation reveals that policies which suppress a task from its initial state can still complete it when taking over partway through execution. Even when the task remains unfinished, the robot may drop a held object, making non-completion an incomplete measure of safety. To address these challenges, we introduce State-Aware Fallback Imitation (SAFI) for tasks where holding provides an appropriate alternative response. SAFI comprises two stages, (1) fallback target construction, which assigns stationary arm commands and consistent gripper targets throughout prohibited-task demonstrations, and (2) joint policy adaptation, which learns these targets alongside the original actions of retained tasks. This supervision teaches an explicit alternative response at intermediate execution states while maintaining the robot's other skills. Extensive experiments with OpenVLA-OFT across LIBERO and BadLIBERO tasks evaluate prohibited completion, physical incidents, and retained-task performance. On the evaluated LIBERO tasks, SAFI reduces aggregate target success after takeover to 1.3%, with no recorded physical incidents or action saturation, while preserving high retained-task success.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.