GPWM: Modeling the World as 3D Particle Motion

Robert Ren, Matthew Bronars, He Zhu, Yikai Tang, Jianing Zhang, Sergey Zakharov, Pavel Tokmakov, Katerina Fragkiadaki

November 2026

Two robot arms folding a garment and pushing objects, from the overview video.

TL;DR A generative transformer predicts how the particles and rigid bodies of a 3D scene move, across rigid, deformable, granular and cloth materials, trained on a 25.5M-frame simulated corpus. It cuts prediction error by 31% on average against the best prior learned simulator, and planning in its latent noise reaches 0.48-0.57 task progress on pushing where action-space search stays at or below 0.01.

What they do

Learned 3D dynamics models are usually deterministic, short-horizon and built for one kind of material. GPWM is a generative world model that represents a scene as tracked particles and rigid bodies and generates their future motion. One architecture covers rigid, elastic, elastoplastic, granular and cloth dynamics, with or without robot actuators. The authors also build ParticleWorlds, 25.5 million simulated frames across six domains, and use the model to plan contact-rich manipulation.

How it works

Representation. Deformable material moves by per-particle displacements, rigid bodies by pose increments applied to fixed surface particles. Rods, grippers and hands use the same geometry, so their motion can be given as a condition or generated jointly with the objects.
Architecture. A multi-resolution transformer pools particles into per-object anchors with PointNet++ layers and runs factorized temporal and spatial attention mostly at the coarse levels. It scales to scenes of up to 12,000 particles.
Long rollouts. Flow matching with diffusion forcing gives each future frame its own noise level. Near frames are committed while far ones stay revisable, so generation streams far past the 16-frame window.
Planning. With the model frozen, a sparse reward on the final object position is optimized over the model’s latent noise, by gradient (LNO) or CEM. The generated actuator motion is executed open-loop in simulation.

Results

Lower error than the strongest baseline in all six domains, 31% on average, e.g. Pushing 23.9 to 13.3 mm and Free Fall 53.0 to 40.0 mm. A single Unified model beats the specialists on Dragging and Pick and Place.
Unseen shapes with 5-7 objects raise error only from 37.5 to 45.2 mm, against 48.5 to 120.9 mm for PhysCtrl.
With the same checkpoint, noise-space planning reaches 0.48-0.57 task progress on Pushing and 0.40-0.56 on Picking, while action-space search stays at or below 0.01.

Limitations

GPWM assumes complete, tracked 3D particle state; recovering it from RGB or RGB-D video is left open. Training data and all planning runs are simulated. The Unified model still trails the specialists on four domains, and latent noise optimization is too slow for fast online control.

Videos

Robot interaction domains: garment folding, a dexterous hand, pick and place with the unified model, and dragging.

Figures

From Figure 1: GPWM learns 3D particle dynamics across six physical domains. Top: samples from the six ParticleWorlds... — From Figure 1: GPWM learns 3D particle dynamics across six physical domains. Top: samples from the six ParticleWorlds domains. Middle: given the observed frames and a noisy future, a U-shaped multi-resolution spatio-temporal transformer denoises the future motion of every particle. Bottom: lower error than the strongest baseline in every domain, rollouts 55x past the 16-frame training window, dynamics simulated from images, and planning via guided diffusion.

From Figure 2: GPWM architecture. Noisy per-particle tokens, embedding displacement, actuator motion, geometry, physi... — From Figure 2: GPWM architecture. Noisy per-particle tokens, embedding displacement, actuator motion, geometry, physical parameters and material, pass through a multi-resolution DiT. PointNet++ layers pool particles into per-object anchors on the way down and interpolate back on the way up. Each block applies temporal attention, 3D relative spatial attention and pointwise feed-forward layers, modulated by the per-frame noise level, and the network outputs clean future displacements.

From Figure 3: Predictions from a single image. Each row shows the input with the reconstructed objects outlined, the... — From Figure 3: Predictions from a single image. Each row shows the input with the reconstructed objects outlined, their particles, and the predicted motion. Top: a tower of a toy car, a pepper, a lemon and a ball with different materials. Bottom: an elastic ball dropped onto a sand or a plasticine castle.

From Figure 4: Planning performance. Task progress versus planning budget on Pushing, Pile Manipulation and Picking, ... — From Figure 4: Planning performance. Task progress versus planning budget on Pushing, Pile Manipulation and Picking, measured after open-loop execution in simulation.

Abstract

We introduce GPWM, a Generative Particle World Model that predicts how 3D scenes evolve by generating the motion of particles and rigid bodies over time. Unlike video world models that generate scene appearance, GPWM operates directly in 3D physical space. A single representation and architecture cover rigid, deformable, cloth, and granular materials, in both passive physics and robotic interaction. GPWM is an object-aware multi-resolution transformer that combines flow matching with diffusion forcing to generate coherent physical trajectories far beyond its training horizon. We train and evaluate GPWM on ParticleWorlds, a 25.5-million-frame corpus spanning diverse geometries, materials, interactions, and robot embodiments. GPWM outperforms the best prior learned-dynamics baseline in each of six physical domains, reducing prediction error by 31% on average. Breaking from the domain-specific setting of prior methods, a single Unified GPWM models these heterogeneous regimes with one set of weights, largely outperforming per-domain baselines and even surpassing our own specialist models in two domains. GPWM further generalizes to unseen geometries, physical properties, object counts, and rollout horizons. Beyond prediction, GPWM enables contact-rich planning by learning a generative prior over joint actuator-object motion. Instead of searching over unconstrained actions, planning searches the model's latent noise for interactions that achieve a desired object-space goal. With the same world model, noise-space planning substantially outperforms action-space search on pushing and picking, where complex contact sequences are hard to discover. Executed open-loop in simulation, the resulting plans achieve goal-directed manipulation without reference action trajectories or goal-specific policies.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong