Be My Eye: Learning to See Beyond a Robot's Own Perspective

Baifu Huang, Shilin Fang, Jianan Wang

November 2026

TL;DR Two robots each compress what their cameras see into 32 task-conditioned tokens and feed them into the partner's pi0.5 policy. On real Astribot S1 robots, choosing a box that only the partner saw being marked rises from 38.9% to 88.9% accuracy, ahead of sharing 160 generic SigLIP tokens (72.2%).

What they do

In multi-robot work, one robot often sees what its partner cannot. Instead of sending raw images or generic visual features, each robot encodes its camera views into a few task-conditioned tokens that the partner’s vision-language-action (VLA) policy takes as extra input. Both robots send and receive, with no fixed roles.

How it works

Encoder. Frozen vision and language encoders produce features; 32 learnable queries pool the multi-camera features into tokens conditioned on the instruction.
Stage 1. The tokens learn to predict future visual features, weighted toward patches a vision-language model rates as relevant to the instruction, plus a soft contrastive loss that groups samples by task and execution progress.
Stage 2. The tokens enter a pretrained pi0.5 policy. The receiver’s action loss backpropagates into the sender’s encoder, so what gets shared is shaped by what the partner needs.

Results

On two real Astribot S1 robots:

Occluded-target selection (18 trials each): accuracy is 38.9% without sharing, 72.2% sharing 160 SigLIP tokens, and 88.9% with 32 learned tokens.
Leader-follower marker retrieval (48 trials): without sharing, no full sequence succeeds; SigLIP sharing and the method both reach 100%, the method sending 144 instead of 720 KiB per step.
Ablations: removing the contrastive loss drops the process score from 100% to 68.9%, freezing the encoder in Stage 2 to 83.3%; 4 tokens still score 90.5%.

Limitations

What the shared tokens encode, and how they shape the receiver’s actions, is not analyzed. Quantitative results cover two robots and two tasks with small trial counts, and in both the robot providing information stays still while its partner acts.

Figures

From Figure 1: Overview of the two-stage framework. In Stage 1, multi-camera observations and the task instruction ar... — From Figure 1: Overview of the two-stage framework. In Stage 1, multi-camera observations and the task instruction are compressed into shared context tokens, trained with weighted future visual feature prediction and soft-label contrastive learning. In Stage 2, the robots exchange their context in both directions and condition their VLA policies on it to generate continuous actions.

From Figure 3: Rollouts of the two evaluated tasks. In occluded-target selection, Robot B uses Robot A's view to plac... — From Figure 3: Rollouts of the two evaluated tasks. In occluded-target selection, Robot B uses Robot A's view to place the toy in the box holding the blue plate; in alternating leader-follower marker retrieval, each robot picks the pen matching its partner's choice.

From Figure 4: Rollouts of three more collaborative tasks: restocking a fridge, assembling a box, and folding and pac... — From Figure 4: Rollouts of three more collaborative tasks: restocking a fridge, assembling a box, and folding and packing a blanket. Keyframes mark moments of context sharing between the robots.

Abstract

Robots operating in the real world are inherently subject to partial observability, particularly in multi-robot collaboration, where individual observations may be insufficient for effective task execution. A key challenge is leveraging complementary observations from partner robots to support collaborative decision-making. We argue that information sharing should go beyond passively transmitting raw observations or generic visual features. Instead, robots should selectively encode task-relevant information according to task objectives and execution progress. To this end, we propose a task-oriented active context sharing framework that transforms robot observations into compact representations, preserving critical information under constrained communication budgets. The framework adopts a two-stage training paradigm. In the first stage, onboard observations and task instructions are jointly encoded into compact context representations. Task-oriented future prediction and soft task-progress correspondence learning guide this collaboration-agnostic compression, preserving information about task objectives and execution progress. In the second stage, the learned representations are integrated into a pretrained Vision-Language-Action (VLA) policy, enabling bidirectional context exchange. Each robot combines complementary partner information with local observations to guide action generation. Action supervision jointly refines the context representations and policy, aligning information sharing with collaborative decision-making. Controlled experiments demonstrate substantial performance gains under constrained communication budgets, while three real-world experiments validate the framework in complex dual-robot tasks. These results highlight a shift from passive visual transmission toward active, task-oriented context sharing that directly supports collaborative decision-making.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong