Physical Diffusion Policy via MPM-Guided 3D Gaussian Simulation

Can Wang, Qiyuan Qiao, Lei Liu, Ge Yuan, Xinhui Liu, Dong Xu

November 2026

TL;DR Estimate each object's stiffness, plasticity and viscosity by fitting a Gaussian-splat physics simulation to a demonstration video, then condition a mixture-of-experts diffusion policy on those parameters. On a real robot, pick-and-place success across 18 everyday objects rises from 28.9% (Diffusion Policy) to 62.5%.

What they do

Diffusion policies see objects only by appearance, so one policy trained on soft, foamy and rigid objects tends to average incompatible behaviors. This paper estimates each object’s material parameters from a demonstration video, using a 3D Gaussian splat simulated with a differentiable Material Point Method (MPM), and conditions a mixture of material-specific diffusion experts on them.

How it works

Physics from video. A feed-forward 3DGS model reconstructs the object and binds each Gaussian to an MPM particle. Young’s modulus, Poisson’s ratio, yield stress and viscosity are optimized until the rendered simulation matches a video of the robot handling the object, starting from a VLM guess, with material-specific bounds, gripper-following particles and a Chamfer geometry term.
Material-aware policy. Four diffusion experts (elastic, foam, plastic, rigid) are pretrained per material, then fine-tuned jointly with a router driven by the physical parameters and camera features.
Memory bank. Optimization takes 798 s per object, so at deployment DINOv2 features retrieve the parameters of the closest stored object in about 415 ms.

Results

Real robot (100 teleoperated demos, 30 trials per task): pick-and-place success averaged over four material classes is 62.5%, vs 28.9% for Diffusion Policy; physics conditioning alone reaches 49.4%. Elastic objects gain most over DP3 (57.1% vs 30.0%).
Simulation (Isaac Lab): 89.9% average pick-and-place success vs 83.3% for Diffusion Policy.
Grapes and paper cups stay intact in 76.7% and 90.0% of trials, vs 43.3% and 73.3% for DP3.
Bank retrieval costs 1.5 points of real-robot success vs online optimization (59.2% vs 60.7%).

Limitations

The paper has no limitations section. Real-robot success per material class stays between 46.7% and 70.0%, and only pick-and-place and stacking are tested. A new object gets the parameters of the most similar object in the bank, and the experts are tied to four fixed material categories.

Figures

From Figure 1: Training pipeline. Multi-view images give a feed-forward 3D Gaussian splat that is simulated with MPM;... — From Figure 1: Training pipeline. Multi-view images give a feed-forward 3D Gaussian splat that is simulated with MPM; its material parameters are optimized through a differentiable rasterizer to match a demonstration video, with gripper boundary conditions and a Chamfer geometry constraint. The estimated parameters condition a mixture-of-experts diffusion policy together with language and camera images.

From Figure 3: Object preservation on fragile objects, where a trial counts only if the grape is not ruptured or the ... — From Figure 3: Object preservation on fragile objects, where a trial counts only if the grape is not ruptured or the paper cup not crushed. Success is 76.7% (grape) and 90.0% (cup), against 33.3% and 46.7% for Diffusion Policy and 43.3% and 73.3% for DP3.

From Figure 2: Task success in simulation as one physical parameter at a time is perturbed away from its true value. ... — From Figure 2: Task success in simulation as one physical parameter at a time is perturbed away from its true value. Success falls as the error grows, most for Young's modulus, yield stress and viscosity and least for Poisson's ratio.

Abstract

Learning robust manipulation policies for diverse everyday objects remains challenging because objects with different materials exhibit distinct deformation and contact dynamics. While diffusion policies provide a powerful framework for modeling multimodal manipulation behaviors, they typically lack explicit physical reasoning and may produce averaged actions when facing heterogeneous objects. We present a physics-aware diffusion policy that augments visual imitation learning with object physical understanding. Our framework recovers material properties from visual demonstrations by coupling 3D Gaussian Splatting with differentiable Material Point Method simulation, where video guidance and particle-level constraints enable accurate physical parameter estimation. The estimated physics conditions are further integrated into a material-aware Mixture-of-Experts diffusion policy, allowing adaptive behavior generation across different physical regimes. For practical deployment, we introduce a physics memory bank that provides efficient physical retrieval without requiring online simulation optimization. Experiments on a wide range of simulated and real-world objects demonstrate improved manipulation success and generalization over existing diffusion-based approaches. These results show that incorporating physical knowledge is critical for scalable robot manipulation in diverse environments.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong