Hybrid Particle-Voxel Dynamics for Multimodal 3D World Simulation

Jimmy Wang, Yifan Chou, Kaifeng Zhang, Yixuan Wang, Binghao Huang, Tao Yu, Saleh Nabi, Yunzhu Li

November 2026

A policy success and a policy failure on T pushing, both generated by the 3D World Simulator.

TL;DR An action-conditioned world model that predicts the scene as 3D voxel latents, decoded into Gaussian particles and tactile points, so it can render any camera view and fingertip touch at 50 Hz. Across six bimanual ALOHA tasks it beats 2D video and 3D particle baselines on every image, depth, geometry and tactile metric, and its generated data trains imitation policies.

What they do

Most action-conditioned world models predict 2D video, which makes it hard to stay consistent across viewpoints. 3D World Simulator predicts how the 3D scene evolves instead. Each state decodes into Gaussian particles that render from any camera, including moving wrist cameras, and into tactile points aligned with the contacted surfaces. They use it to generate policy training data, evaluate policies and plan.

How it works

Hybrid particle-voxel autoencoder. Stereo point clouds, plus FlexiTac tactile pads placed by forward kinematics, are compressed with sparse 3D convolutions into a voxel latent. The decoder upsamples it to finer voxels, then to dense particles carrying Gaussian parameters.
3D latent dynamics. A 3D U-Net with spatial and causal temporal attention predicts the next latent from the history and the action. It is trained on multi-step rollouts with a horizon curriculum and noise added to the history latents.
Composition. The output can be combined with Gaussian backgrounds and kinematic robot models, including arms not seen in training.
Data. About 200 teleoperated episodes (2 h) per task on a bimanual ALOHA, plus the Deform360 visuo-tactile dataset (198 objects, 215 h).

Results

Averaged over six ALOHA tasks and 200-step open-loop rollouts, it beats the best baselines on every metric: PSNR 23.24 vs 20.59, scene Chamfer 7.72 vs 10.43, tactile L1 0.081 vs 0.128.
On Deform360 its Chamfer distance is 9.6 mm on training objects and 12.1 mm on held-out objects, vs 10.5 and 13.4 mm for the best particle baseline. Held-out rendering is worse (PSNR 14.8 vs 15.6).
With 100 demonstrations per task, Part Assembly policies trained on generated visuo-tactile data match those trained on real data and substantially beat vision-only real-data policies. Success rates in the simulator track real success rates for ACT, DP, π0.5 and Cosmos Policy.
The Deform360 model plans zero-shot with MPPI on four unseen objects and an unseen ALOHA arm.

Limitations

The model covers one cropped workspace; larger scenes need composed Gaussian assets. On held-out objects it renders worse than particle baselines that track fitted splats. The Deform360 model sees only object points, not the robot or the scene.

Videos

The raw autoencoder output composed with a static 3D Gaussian background and a kinematic robot model.

The same box scene rendered with swapped backgrounds and robot arms.

Figures

From Figure 1: Overview of 3D World Simulator. An action-conditioned, multimodal 3D world model that jointly predicts... — From Figure 1: Overview of 3D World Simulator. An action-conditioned, multimodal 3D world model that jointly predicts 3D geometry (middle), multi-view consistent RGB and depth from novel views (upper left), and spatially aligned tactile signals (bottom left), across six tasks.

From Figure 2: Model architecture. (a) A hybrid particle-voxel autoencoder compresses an input point cloud into a com... — From Figure 2: Model architecture. (a) A hybrid particle-voxel autoencoder compresses an input point cloud into a compact 3D voxel latent, then upsamples it into finer voxels and dense visual and tactile particles; each visual particle carries Gaussian parameters for rendering at novel viewpoints. (b) The 3D latent dynamics model uses 3D convolutions and spatiotemporal attention to predict future voxel latents from robot actions and history latents.

From Figure 6: Policy training on 3D World Simulator and real-world data. Imitation learning policies trained purely ... — From Figure 6: Policy training on 3D World Simulator and real-world data. Imitation learning policies trained purely on generated data perform comparably to policies trained on real data, across visuo-tactile, multi-view and novel-view observations.

From Figure 7: Correlation between simulation and real-world policy performance. Policies evaluated in 3D World Simul... — From Figure 7: Correlation between simulation and real-world policy performance. Policies evaluated in 3D World Simulator and in the real world show strong correlation across all three tasks.

Abstract

Predicting how the world evolves in response to robot actions is central to policy learning, evaluation, and planning. However, many action-conditioned world models predict future observations in 2D image or video space without explicitly modeling the underlying 3D scene, making spatial consistency across viewpoints and over long-horizon interactions challenging. We introduce 3D World Simulator, an action-conditioned multimodal world modeling framework that jointly predicts appearance, geometry, and tactile observations from a unified 3D representation. Our approach combines a hybrid particle-voxel autoencoder that compresses point clouds into compact 3D voxel latents with an action-conditioned latent dynamics model that predicts their evolution. The shared 3D representation supports geometrically consistent rendering from arbitrary viewpoints and flexible integration of spatially aligned tactile signals. Our experiments demonstrate interactive, long-horizon, multimodal prediction at 50 Hz. We then show how our model enables scalable data generation for policy training and evaluation using tactile and multi-view visual observations, including wrist-mounted and novel camera views. Furthermore, we scale our method to large datasets containing diverse objects and demonstrate model-based planning with unseen objects and robot configurations. Together, these results highlight the potential of hybrid particle-voxel dynamics as a foundation for multimodal 3D world simulation.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong