← Paper Digest

Learning to Move from and for Manipulation Policies: Spatializing Latent Action Representations for Goal-Pose-Free Mobile Manipulation

Siyuan Zheng, Zhong Cao, Henry X. Liu

November 2026

Rollout in a held-out simulated kitchen, sped up 10x.

TL;DR M4M reads the hidden states of a frozen manipulation policy such as π0.5 to predict, from the robot's current stereo view, how likely the policy is to succeed from each nearby base pose, then moves the base and hands off when success looks high. Across eight articulated-object tasks in unseen simulated kitchens it reaches 47.0% mean success, versus 26.5% for N2M and 19.2% for executing directly.

What they do

A manipulation policy that works from a good stance often fails when a mobile robot starts 0.5-2.5 m away or at the wrong angle. M4M keeps the manipulation policy frozen and learns where to move the base and when to hand off to it. Instead of navigating to a fixed goal pose, it predicts a success field over candidate stances from the current view and moves along it.

How it works

  • Teacher from policy features. In simulation, frozen π0.5 is run (without executing actions) on the view from each candidate stance. Its final action-expert hidden states, compressed by PCA to 128 dimensions, feed a logistic probe fitted on a limited budget of executed rollouts, giving dense success scores without rolling out at every stance.
  • Current-view student. A ResNet-18 Lift-Splat backbone and the policy’s own visual tokens build bird’s-eye geometry, target and traversability maps from stereo. A cross-attention decoder predicts each candidate’s features from the current pose, trained with feature matching, score matching and rollout outcomes.
  • Move and hand off. A small CNN controller, distilled from value iteration on the simulator grid, chooses among 16 directions or stops; moves are 0.15 m and collision-checked against predicted traversability.

Results

  • Eight articulated-object tasks in RoboCasa, two held-out kitchens each, 1,000-rollout budget: 47.0% mean success vs 26.5% for N2M and 19.2% for direct π0.5. The one-shot variant without navigation reaches 33.3%.
  • Refrigerator closing with four frozen policies: 82.2% vs 51.7% for N2M (π0.5), 55.0% vs 14.4% (π0), 55.6% vs 16.1% (GR00T N1.5), 23.3% vs 12.2% (Diffusion Policy).
  • With a box blocking the approach: 47.2% vs 29.2% for N2M. On the same fields, the learned controller beats Dijkstra movement (82.2% vs 72.8%).
  • On unseen layouts, action-expert hidden states rank stances better than ResNet-18, SigLIP tokens, the fused prefix or the decoded actions.

Limitations

Simulation only, with 0.5-2.5 m starts and a separate predictor per task; cross-policy tests cover only refrigerator closing. Training uses privileged geometry and rendered candidate views, so real-world supervision would need candidate-view capture or scene reconstruction. Physical robots, longer-range navigation and moving obstacles are untested.

Videos

Animation of the robot scoring candidate stances from its current view while it approaches the refrigerator.
Overview: from stereo views and "Close the fridge door", the robot must find the target, get around obstacles and know when to hand off.

Figures

From Figure 2: In training, the frozen policy observes a candidate stance and its action-expert features give a probe...
From Figure 2: In training, the frozen policy observes a candidate stance and its action-expert features give a probe success score; a student learns to predict those features from stereo and language at the current pose. A compact controller reads the resulting success, target and traversability fields to choose collision-checked moves and decide when to hand off.
From Figure 3: Four frozen policies (π0.5, GR00T N1.5, π0 and Diffusion Policy) evaluated at the same 100 stances aro...
From Figure 3: Four frozen policies (π0.5, GR00T N1.5, π0 and Diffusion Policy) evaluated at the same 100 stances around a refrigerator. Color is empirical success from 0 to 1; each policy prefers different stances.
From Figure 5: Success on eight tasks in two held-out kitchens (180 episodes per method) for direct execution, N2M, M...
From Figure 5: Success on eight tasks in two held-out kitchens (180 episodes per method) for direct execution, N2M, M4M one-shot and M4M closed-loop. Dashed lines mark success from a favorable stance.
Abstract

The success of manipulation policies is closely tied to the spatial relationship between the robot and the target object, as well as the task being performed, making the ability to adjust the robot's pose critical for successful execution. When the robot starts away from the task, such adjustments require coordinating navigation with local manipulation. Jointly training these capabilities incurs substantial computational and data costs, while navigating toward a predefined goal pose limits the flexibility needed to adapt to task-specific manipulation requirements. To address these challenges, we propose Learning to Move from and for Manipulation Policies (M4M), a framework that learns manipulation-aware mobility directly from pretrained manipulation policies. By leveraging the hidden representations of the policy's action-generating network as dense supervision, together with a limited number of manipulation execution outcomes, we learn a spatially conditioned embedding that captures task-dependent manipulation success across candidate robot poses. From the robot's current stereo observations, this representation estimates manipulation success at unvisited poses without rolling out the manipulation policy at each location. It is further integrated into navigation policy training, while a learned handoff policy decides when to transition from navigation to manipulation, enabling the robot to move and manipulate in a unified, task-aware manner. Across eight articulated-object tasks in unseen RoboCasa kitchens, M4M achieves 47.0% mean task success, compared with 26.5% for the strongest baseline and 19.2% for direct manipulation execution, and refrigerator-closing experiments show consistent improvements across four frozen manipulation policies.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.