ObjectWorld: Scaling Object-Centric World Models with A Large-Scale 3D Full-Object Flow Dataset

Haonan Chen, Sicheng Isabella Lu, Haitian Li, Jiajin Wu, Hexiang Wei, Jiawei Zhang, Xin Fei, Yiwen Hou, Chenrui Tie, Chongkai Gao, Sheng Wang, Lin Shao

November 2026

Real-world experiments: predicted full-object flow and the robot executing banana, pouring, flashlight and cabbage tasks, 4x speed.

TL;DR Predict how every point on an object's full surface, hidden sides included, should move, and let a small optimizer turn that into gripper motion. Trained on 1,210 hours of auto-annotated video and simulation, the model roughly halves final-frame flow error against baselines (11.17 cm vs. 20.88 cm) and succeeds in 18/20 real-robot trials.

What they do

World models for manipulation need an output a robot can act on. ObjectWorld proposes 3D full-object flow: dense 3D trajectories for points over the complete surface of the target object, including the occluded side, for rigid and deformable objects alike. The authors build a 1,210-hour dataset with this annotation, train a model that predicts the flow from one RGB-D image and an instruction, and execute it on robots.

How it works

Annotation pipeline. From plain RGB video (no depth, calibration or meshes), Gemini 2.5 Pro names the target and cuts atomic clips, Track4World recovers cameras and point clouds, SAM 3D reconstructs the full mesh, and Any6D plus FoundationPose track its pose. Points sampled on the mesh give the flow; clips with low reprojection IoU are filtered out.
Dataset. 454.5 h from seven simulation benchmarks and 755.6 h of real video (AgiBot World, Something-Something V2, EPIC-KITCHENS); the 288.7 h “Golden” real subset is used for training.
Predictor. Frozen Qwen2.5-VL features and a joint Point Transformer V3 over object and scene points feed a decoder that denoises 48-step trajectories for 512 object points.
Execution. After a grasp, the end-effector pose at each step is fit to the predicted motion of a local contact patch.

Results

Annotation, checked on 97 RoboTwin clips with ground truth: median flow error 3.71 cm. The Golden filter reaches 93.3% precision on 200 expert-labeled real clips.
Flow prediction: final-frame error 11.17 cm, against 20.88 to 26.43 cm for im2Flow2Act, ObjectForesight and TraceGen. Error generally falls as simulated or real data is added.
Simulation execution (ManiSkill, LIBERO): 39/40 across four tasks, against 26/40 for ObjectForesight.
Real robot: 4/5 banana, 4/5 pouring, 5/5 flashlight with a parallel gripper and 5/5 cabbage with a dexterous hand; Dream2Flow gets 2/5, 3/5, 5/5 and 4/5.

Limitations

The paper has no limitations section. From its own numbers: the annotated flow still has a median error of 26.11% of object diameter on simulated clips; real-video annotations come from 6-DoF pose tracking of a reconstructed mesh; execution assumes a locally rigid contact patch and a separate grasp module; and each real-world task is evaluated with five trials.

Videos

Simulation experiments on in-domain and out-of-domain tasks.

Full-object flow annotations recovered from real-world videos in the dataset.

Figures

From Figure 2: Automatic data annotation pipeline. Given an in-the-wild manipulation video, a VLM identifies the targ... — From Figure 2: Automatic data annotation pipeline. Given an in-the-wild manipulation video, a VLM identifies the target object, describes the task, and segments the video into atomic action clips. For each clip, the pipeline extracts the target-object masks, reconstructs the complete object mesh and the point clouds, and aligns the mesh with the point clouds across frames to recover the object motion. The motion is propagated over the complete object surface to produce 3D full-object flow.

From Figure 3: Object-centric full-object flow prediction framework. Given an initial RGB-D observation and task inst... — From Figure 3: Object-centric full-object flow prediction framework. Given an initial RGB-D observation and task instruction, Qwen2.5-VL extracts task-conditioned visual-language features, while a joint PointTransformer V3 (PTv3) encodes reconstructed object and scene geometry. These features are fused into object tokens and refined by self-attention. The object queries attend to scene and language context through a cross-attention decoder, conditioned on noisy flow and time, before a flow head predicts dense trajectories over the complete object surface.

From Figure 5: Qualitative full-object flow predictions. Given an initial RGB observation and task instruction (top),... — From Figure 5: Qualitative full-object flow predictions. Given an initial RGB observation and task instruction (top), the model predicts dense 3D trajectories over the complete target-object surface (bottom) across real-world and simulated tasks.

From Figure 6: Data scaling. Final-frame endpoint error (EPE, lower is better) of the flow predictor for different fr... — From Figure 6: Data scaling. Final-frame endpoint error (EPE, lower is better) of the flow predictor for different fractions of simulated and real-world training data.

Abstract

Generative world modeling from large-scale visual data offers a promising path toward scalable robot manipulation. This calls for an intermediate representation that can be supervised at scale and remains actionable for downstream control. We propose 3D full-object flow, an object-centric motion representation that assigns dense 3D trajectories to points across the complete surface of a target object, including its occluded regions. By extending visible-surface object flow to the full object geometry, this representation makes complete object motion explicit and applies uniformly to both rigid and deformable objects. To realize the potential of this representation at scale, we introduce a 1210-hour full-object flow dataset that addresses the scarcity of large-scale supervision for complete 3D object motion. The dataset integrates diverse real-world videos and physical simulations, with a scalable annotation pipeline for recovering full-object motion from real-world observations and quality metrics for systematically assessing annotation quality. We then train a world model that takes an image and a task description as input and predicts dense full-object flow over the reconstructed object geometry. Experiments show reliable annotation quality, data-scalable flow prediction, and effective downstream manipulation across novel scenes and robot embodiments. Project website with additional results and videos: https://nus-lins-lab.github.io/ObjectWorld/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong