I See What You Want: Decoding Hierarchical Operator Intent from Visual Teleoperation History

Xiangyi Jia, Kaixian Qu, Shengzhi Wang, Chenyu Yang, Robert Katzschmann, Marco Hutter

November 2026

Dataset episode from the project page, a Franka arm at a table with a tomato, a pot and a box.

TL;DR A VLM fine-tuned on VI AMO, 6,600 robot episodes labeled with atomic actions and the missions they serve, guesses from a few teleoperation frames what the operator is doing and why, at 0.825/0.821 SBERT similarity versus at most 0.581/0.564 for GPT and robotics VLMs. Handing its predicted mission to a VLA policy on a real Franka cut teleoperation time by 34.6% compared with using GPT-4o.

What they do

Can a model watching a teleoperated robot tell what the operator is doing and why, with no instruction given? The authors split intent into the ongoing action (“pick up the apple”) and the mission it serves (“put the apple on the plate”), both as free-form text. They build a dataset for this, fine-tune a VLM on it, and use the predicted mission as the instruction for a VLA policy that takes over mid-task.

How it works

VI AMO. 6,600 episodes from eleven sources, among them Bridge V2, DROID and RT-1. GPT-5.6 Sol splits each instruction into atomic actions and groups them into missions; Qwen3.7-Plus localizes action boundaries, refined with gripper events; humans reviewed about half the episodes.
VIM. Qwen3.5-27B with LoRA takes four frames per camera plus a list of completed actions and outputs action and mission in one pass. Training samples partial windows of each action so it learns to answer early.
Shared autonomy. On a Franka teleoperated with a Rokoko glove, VIM proposes a mission, the operator confirms it, and a fine-tuned pi0.5 policy continues.

Results

On 550 held-out episodes VIM scores 0.825/0.821 SBERT (action/mission) and 76.59%/78.77% top-10 retrieval, versus at most 0.581/0.564 and 39.14%/39.33% for GPT-4o, GPT-5.6 Luna, the untuned backbone, RynnBrain and RoboBrain.
It already reaches 0.776/0.776 at 0% of action progress, and 0.726/0.717 on the 35.4% of test actions whose labels never appear in training (best baseline 0.570/0.564).
Dropping wrist views, completed-action context or frame history lowers action SBERT to 0.798, 0.767 and 0.793.
Real robot, five tasks, five trials each, in scenes unseen during fine-tuning: 34.6% less teleoperation than with GPT-4o and 39.8% less than with GPT-5.6 Luna, at 0.70-0.81 s per prediction versus 1.92-3.53 s. On trash sweeping neither GPT model predicted a usable intent.

Limitations

VIM still makes errors and has no calibrated confidence, so when to trust it is open. The benchmark assumes given action boundaries rather than detecting them live. The VLA is slower than a skilled operator, so less teleoperation does not always mean faster completion.

Videos

Dataset episode from the project page, a robot arm wiping a table with a cloth.

Dataset episode from the project page, a robot arm moving a bottle next to a bowl.

Figures

From Figure 2: (a) Annotation pipeline. A VLM splits each episode instruction into ordered atomic actions and groups ... — From Figure 2: (a) Annotation pipeline. A VLM splits each episode instruction into ordered atomic actions and groups them into the missions they serve; coarse-to-fine video segmentation localizes each action in time, followed by human review. (b) The VI AMO dataset: 6,600 episodes from eleven robot sources, 13,653 actions and 9,702 missions. (c) VIM is fine-tuned to generate the ongoing action and mission from a short visual history and the completed-action context.

From Figure 6: Handoffs on the real robot. Top, left to right: the shared-autonomy setup, egg placement and trash swe... — From Figure 6: Handoffs on the real robot. Top, left to right: the shared-autonomy setup, egg placement and trash sweeping. Bottom: three successive missions in one rollout, placing a cherry, an apple and a banana on a plate. Bars mark intervals of human teleoperation and VLA control and the active mission.

From Figure 5: Real-world results per task, mean and standard deviation, for full teleoperation and for shared autono... — From Figure 5: Real-world results per task, mean and standard deviation, for full teleoperation and for shared autonomy with GPT-4o, GPT-5.6 Luna or VIM: (a) human teleoperation time; (b) VLM latency per request.

Abstract

Vision-language models (VLMs) have become a powerful foundation for robotic scene understanding, task planning, and language-conditioned control. A natural question follows: can these models also decode what a human operator intends while teleoperating a robot, without any explicit instruction? We formulate this as hierarchical intent prediction, jointly inferring the ongoing atomic action and the broader mission it serves from a short visual teleoperation history. To study this, we introduce VI AMO (Vision Intention Action and Mission Ongoing), a dataset of 6,600 episodes from eleven robot sources with 13,653 temporally aligned action annotations and 9,702 mission annotations. Large proprietary models struggle with this problem: on held-out episodes, GPT-4o reaches Sentence-BERT similarities of only 0.563 and 0.470 for action and mission, and GPT-5.6 Luna achieves only 0.581 and 0.554. Fine-tuning a VLM on VI AMO yields our Vision-Intention Model (VIM), which raises these scores to 0.825 and 0.821 and still reaches 0.726 and 0.717 on the 35.4% of test actions with both labels absent from training. Finally, as a downstream application, VIM's predicted mission enables a vision-language-action (VLA) policy to take over from the operator mid-task in a real-world environment unseen during VIM fine-tuning, reducing human teleoperation time by 34.6% relative to GPT-4o and 39.8% relative to GPT-5.6 Luna, while cutting intent-prediction latency by roughly 70% on average.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong