PointPilot: Unified Robotic Manipulation via Object-Centric 3D Point Flow Prediction

Yihan Fang, Yuxin Tian, Ruixiang Wang, Zhanqi Zhang, Xing Hu, Zhen Yang, Huayi Zhou, Yongyi Su, Wei Gao, Kui Jia

November 2026

TL;DR PointPilot predicts the 3D point flow of whatever matters in the current subtask (the gripper before a grasp, the held object after) and turns it into end-effector poses with a non-learned rigid fit. One policy reaches 85.8% on LIBERO and 56.6% on RoboTwin, transfers zero-shot to unseen arms in simulation, and succeeds in 42 of 60 real bimanual trials.

What they do

PointPilot is a manipulation policy whose output is the motion of 3D points on a “task-focused object”: the gripper itself while reaching, then the object rigidly held by it after the grasp. Because it predicts object motion rather than joint commands, demonstrations from different robots share one supervision space, and a single model handles single-arm and bimanual tasks.

How it works

Planning. A Qwen2.5-VL planner reads the main camera image, the instruction and a memory of earlier predicted subtasks, and outputs the current and next subtask for each arm.
3D grounding. Image tokens get 3D positional embeddings from camera frustums, with no depth input. A transformer decoder predicts a 3D box for each task-focused object and the gripper it belongs to.
Point Flow DiT. Points sampled in each box are denoised with flow matching into 30-step future displacements, together with gripper open/close states. Auxiliary camera views are aligned into the main-camera frame.
Execution. A learning-free adapter fits a rigid transform to the predicted flow and applies it to the assigned end effector through IK. The policy replans after every 30-step chunk.

Results

One checkpoint trained jointly on LIBERO and RoboTwin scores 85.8% on LIBERO (96.0% Spatial, 97.6% Object, 77.2% Long). On RoboTwin 2.0 Clean it reaches 56.6%, against 55.2% reported for DP3.
Zero-shot transfer from ALOHA to unseen arms in RoboTwin costs 10.4 points of average success on ARX and 16.6 on Piper. X-VLA loses 11.4 and 64.7.
On a real CobotMagic bimanual robot (50 demonstrations, 20 trials per task): QR-code scanning 10/20, cube-to-basket 18/20, pouring 14/20.
Without planning memory, LIBERO average falls to 75.2% (LIBERO-Long to 55.0%).

Limitations

The method assumes locally rigid objects and gripper-based execution; dexterous hands and human-hand data are left to future work. Only the current subtask’s objects are predicted, not other objects they affect. Three friction-driven LIBERO-Goal tasks, where the object is not coupled to the gripper, pull its score down. Cross-embodiment transfer is shown only in simulation, and the real-world policies are trained per task.

Figures

From Figure 1: Given a language instruction and multi-view RGB images, PointPilot predicts the 3D point flow of the t... — From Figure 1: Given a language instruction and multi-view RGB images, PointPilot predicts the 3D point flow of the task-focused object in each subtask, first the gripper while reaching, then the green block while stacking. A learning-free geometric adapter and IK turn the flow into end-effector commands. Right: single-arm, bimanual, real-world and zero-shot cross-embodiment examples.

From Figure 2: (a) Planning: a memory-conditioned VLM predicts the current and next subtasks. (b) Grounding: task-con... — From Figure 2: (a) Planning: a memory-conditioned VLM predicts the current and next subtasks. (b) Grounding: task-conditioned 3D grounding localizes the task-focused objects and assigns each to a gripper. (c) PointFlow: geometrically aligned multi-view observations condition a Point DiT that jointly predicts relative 3D point flow and gripper states.

From Figure 3: Autonomous closed-loop real-world manipulation: (a) QR-code scanning, (b) cube-to-basket transfer, (c)... — From Figure 3: Autonomous closed-loop real-world manipulation: (a) QR-code scanning, (b) cube-to-basket transfer, (c) pouring. Each row shows successive left- and right-arm subtasks with the predicted point flows.

Abstract

Generalization remains a central challenge in robotics, calling for policies that address domain variations from multiple dimensions such as diverse embodiments, camera poses, and/or scene layouts. In order to approach this problem, we introduce PointPilot, a task-focused and 3D aligned manipulation framework built on a 3D space point-flow prediction. Rather than modeling the global scene, PointPilot only represents task-focused objects (robot grippers treated as objects), avoiding the prediction of potentially redundant scene parts unrelated to the manipulation tasks. Moreover, PointPilot aligns 3D spatial representation in one unified coordinate and supports closed-loop and fully automatic action execution across different manipulation modes from object point flow without a separately trained action expert, while existing works typically need an extra action head or are (semi-)automatic and single-arm mode specific. Given sensor observations and language instructions, our model jointly learns subtask planning, task-relevant object grounding, and 3D point flow prediction in an end-to-end way. By processing heterogeneous data in a shared 3D point flow space, PointPilot is able to absorb demonstrations from different embodiments. Extensive simulative and real world experiments validate our method. Notably, we demonstrate zero-shot embodiment generalization with PointPilot. A single trained policy can be deployed on previously unseen embodiments and successfully complete manipulation tasks without any additional fine-tuning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong