MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes

Richard Shi, Mohammad Mahdavian, Mehrdad Momeni Zadeh, Mohammed Elmahgiubi, Amir Rasouli

November 2026

Real UR5e putting a cup on a coaster, shown with the predicted masks and three camera views.

TL;DR Add-on modules for a pretrained π0.5 VLA that predict target, destination and obstacle masks, inject object-level depth, and steer attention toward nearby obstacles. On the cluttered RoboPRO benchmark success rises from 52.7% to 57.8%, and under visual perturbations from 65.2% to 74.9%.

What they do

VLAs such as π0.5 act from 2D images and language, so in clutter they must infer obstacle geometry implicitly, which can lead to collisions. MOSAIC adds modules to a pretrained VLA that make the target object, its destination and the surrounding obstacles explicit, without changing the base architecture.

How it works

Role masks. Learnable role tokens in the VLM prefix (4 each for target, destination and obstacles) feed a mask head that predicts three masks on the main camera image. The action expert attends to these tokens.
Object depth tokens. A frozen VGGT-Ω supplies depth and confidence. The target and destination masks pool it into a statistics token plus a 4x4 grid of depth tokens per role, read only by the last six action-expert layers.
Obstacle attention. Attention targets come from the distance between points along the robot links and object bounding boxes over a 7-step window. A KL loss steers the first three action-expert layers toward nearby obstacles, and the resulting attention is re-injected into the last three.

Results

RoboPRO (four furnished simulated scenes, 6-15 distractors when cluttered): success goes from 66.0% to 70.1% in clean scenes and from 52.7% to 57.8% in clutter, with the largest gain in cluttered Office (+9.9 points).
Under lighting, blur and pixel-shift perturbations (12 tasks, 20 seeds each), success rises from 65.2% to 74.9%, and in Office from 51.7% to 76.7%. Collision rate stays nearly unchanged.
Dropping the obstacle masks lowers cluttered success from 55.4% to 52.7%.
Real robot: a UR5e trained on 1,000 episodes across five tasks with up to four distractors. Results are shown qualitatively.

Limitations

In clutter the collision rate rises from 39.7% to 44.8%, and collision-free success improves only from 39.7% to 41.3%. The authors attribute this to the policy attempting more manipulations. The real-robot tests report no success rates. Code and data will be released on acceptance.

Videos

Repeated real-robot runs of the cup-on-coaster task, at 4x speed.

Simulated RoboPRO Office and Study tasks where the π0.5 baseline fails and MOSAIC succeeds.

Figures

From Figure 1: MOSAIC overview. A pretrained VLA is augmented with role-specific masks, object-level depth tokens and... — From Figure 1: MOSAIC overview. A pretrained VLA is augmented with role-specific masks, object-level depth tokens and obstacle-focused attention guidance.

From Figure 2: Model structure. (a) The vision-language backbone encodes observations, instruction and robot state, w... — From Figure 2: Model structure. (a) The vision-language backbone encodes observations, instruction and robot state, with role tokens for mask prediction. (b) The action expert attends to the shared prefix and produces attention maps. (c) Object-level depth from VGGT-Ω and the predicted masks becomes tokens injected into the final action-expert layers. (d) Contact-relevant targets supervise attention during training. (e) Predicted attention maps are injected into late action layers. (f) Flow matching generates the actions.

From Figure 6: Real-world experiments. The policy performs different tasks while generating the masks and conditionin... — From Figure 6: Real-world experiments. The policy performs different tasks while generating the masks and conditioning on them during action generation.

Abstract

Vision-Language-Action (VLA) models have demonstrated strong performance in robotic manipulation, yet most rely primarily on 2D visual observations and language instructions, leaving scene geometry and obstacle relationships to be inferred implicitly. This limitation is particularly problematic in cluttered environments, where small spatial errors can lead to unintended contacts or collisions. We propose MOSAIC, a modular spatial- and obstacle-aware framework that can be integrated with pretrained VLA policies to improve object-level scene understanding and action generation. Our method augments the VLA backbone with an auxiliary semantic grounding task that predicts role-specific masks for the target object, destination region, and surrounding obstacles. We further combine the target and destination masks with depth estimated from 3D scene reconstruction to construct role-specific geometric tokens that are injected directly into the action expert. In addition, we propose obstacle-aware attention supervision that emphasizes obstacles near the robot body during task execution. The resulting attention features are subsequently injected into later action-expert layers to further guide action generation. Together, these components provide explicit semantic, geometric, and obstacle-aware cues while preserving the underlying pretrained VLA architecture. We evaluate our framework on pretrained VLA model using the RoboPRO benchmark, including visual perturbation tests, and further validate it on real-world manipulation tasks. MOSAIC improves the success rate by 5.1% in cluttered scenes, and by 9.7% under visual perturbations, with gains of up to 25.0% in the most challenging environment. We will release our code and dataset upon acceptance at the following Github page: https://mmahdavian.github.io/MOSAIC/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong