PAIR: Proprioception-Aligned Image-Ray Representations for Viewpoint-Robust Manipulation

Sung-Wook Lee, Xuhui Kang, Dylan Losey, Yen-Ling Kuo

November 2026

TL;DR PAIR feeds camera rays into the vision encoder of an imitation policy and adds a contrastive loss that pulls together observations with similar robot proprioception, so features stay consistent when the camera moves. On a real Franka with cameras repositioned every rollout, final task success rises 26.7 points over plain behavior cloning.

What they do

An image is a viewpoint-dependent projection of the scene: the same robot state looks different from two cameras, and different states can look alike. PAIR trains the vision encoder of an imitation policy so that observations with similar robot states get similar features regardless of camera pose. It gives the encoder the camera geometry and aligns the resulting image-ray features with proprioception.

How it works

Ray positional encoding. For every visual token, Plücker camera rays are computed relative to both the end-effector and the robot base, encoded by small convolutional networks, and added to the SigLIP position embedding through one identity-initialized linear layer.
Proprioceptive alignment. A supervised contrastive (weighted InfoNCE) loss uses the distance in end-effector position, orientation and gripper width between samples in a batch: the closest pairs are positives, with weights falling linearly to zero at a percentile cutoff.
Training. The loss is added to behavior cloning. It trains a U-Net Diffusion Policy with a SigLIP2 encoder and fine-tunes SmolVLA; with multi-view data, positives are drawn across views.

Results

Diffusion Policy on MimicGen and ALOHA cube transfer with varied cameras: PAIR with rays improves maximum success on average by 16.2 points over BC without rays, 9.1 over ray-conditioned BC and 9.5 over PAIR without rays.
SmolVLA on LIBERO, 50 unseen camera poses per task: 88.9% average against 83.3% for ray-conditioned BC; the Long suite goes from 66.9% to 82.1%.
Real Franka Research 3, three tasks, 30 rollouts each with cameras repositioned every rollout: 76.7% subtask and 57.8% final success, 23.3 and 26.7 points above BC without rays.
Against a time-contrastive (TCN) auxiliary loss, PAIR needs 58.8% fewer FLOPs.

Limitations

The method depends on viewpoint diversity in the training data, and existing robot datasets often cover few or skewed camera poses. Collecting demonstrations from varied, accurately calibrated viewpoints takes extra human effort. The alignment loss adds training cost over plain BC. Without ray conditioning, proprioceptive alignment alone gave mixed gains.

Figures

From Figure 2: Overview of PAIR. (A) PAIR introduces two ray representations: camera rays expressed relative to the m... — From Figure 2: Overview of PAIR. (A) PAIR introduces two ray representations: camera rays expressed relative to the moving end-effector frame and the robot-base frame, both encoded using Plücker coordinates. The ray features are injected into the image features through an identity-initialized linear projection, which acts as a selective positional embedding for each vision token. (B) The joint image-ray representations are aligned according to proprioception similarity using supervised contrastive learning, which induces invariance against viewpoint variation.

From Figure 3: Simulated benchmarks and real-world experiments. In simulation, MimicGen and LIBERO use a Franka Panda... — From Figure 3: Simulated benchmarks and real-world experiments. In simulation, MimicGen and LIBERO use a Franka Panda arm with two side views and one front view, respectively, while AlohaGym uses a bimanual Aloha platform with two front views. In the real world, we evaluate three tabletop tasks using a Franka Research 3 (FR3) arm with two front view cameras, where the camera poses are randomized and estimated using AprilTags at the start of each rollout.

From Figure 4: Success rates (%) of BC (red) and PAIR (blue), with and without ray conditioning, in three real-world ... — From Figure 4: Success rates (%) of BC (red) and PAIR (blue), with and without ray conditioning, in three real-world tasks. Both subtask completion rates and final task success rates are reported.

From Figure 6: Rollout attention maps of SmolVLA models trained on the LIBERO dataset, overlaid on the corresponding ... — From Figure 6: Rollout attention maps of SmolVLA models trained on the LIBERO dataset, overlaid on the corresponding input images. Without ray input, attention is concentrated on spatially irrelevant regions, even with proprioceptive alignment. In contrast, ray-conditioning leads to stronger attention to the end effector, with proprioceptive alignment further concentrating activation around it and its nearby workspace.

Abstract

While imitation learning has made substantial progress toward generalizable robot manipulation through scaling, viewpoint robustness remains a major bottleneck. Visuomotor policies rely heavily on visual inputs, yet images provide an inherently ambiguous signal under viewpoint change: similar scenes may appear markedly different across viewpoints, while distinct states may share certain visual overlaps. A policy must therefore infer not only the task-relevant state, but also how it is visually projected under a particular camera configuration. This entanglement of viewpoint-dependent appearance and task-relevant state can make learning inefficient and limit generalization to unseen viewpoints. Recent work has begun to address this challenge through camera-ray conditioning and representation learning using geometric cues to induce invariance. However, providing camera information as additional context does not by itself ground visual representations in how the task scene is visually projected, while geometric alignment without camera context can introduce spurious correspondences by ignoring viewpoint-dependent projection. We propose PAIR, a contrastive learning framework that aligns joint image-ray representations using robot proprioception. PAIR encourages observations with similar robot states to remain close in representation space across viewpoints, providing a structured grounding signal for learning viewpoint-robust features. PAIR is a highly adaptable framework: it can be used to train Diffusion Policy (DP) or post-train vision-language-action (VLA) models. In simulated and real-world manipulation benchmarks with camera variation, PAIR consistently outperforms the behavior cloning baseline without ray conditioning by 12.7% and 26.7% points, respectively, demonstrating stronger generalization to unseen viewpoints. Project website: PAIRrobot.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong