SoftPAT3D: Physics-Augmented Text-to-3D Scene Generation with Deformables

Lei Shu, Alberta Longhini, Leonidas Guibas, Gordon Wetzstein, Jeannette Bohg

November 2026

TL;DR A text-to-3D scene pipeline whose physics stage simulates cloth, pillows and ropes instead of treating them as rigid, by building each object's rest shape from how it is manufactured. Across 65 prompts no cloth intersects its support, and 21 of 34 scenes pass a completion check against at most 8 for any baseline.

What they do

Text-to-3D scene pipelines settle their layouts with a physics engine, but every object in that stage is rigid. A generated tablecloth or pile of clothes arrives already draped, as a closed and sometimes self-intersecting surface with no rest shape, no material and no valid starting state, so it ends up floating above or cutting into the furniture. SoftPAT3D builds those missing inputs on top of the PAT3D pipeline and simulates rigid supports, cloth and stuffed objects together to static equilibrium.

How it works

Rest shapes from manufacturing. Garments are sewn from parametric sewing patterns, linens are flat panels, and pillows are two seamed panels with a tetrahedralized interior, each clamped to the real-world size range of its class.
Initial state. The cloth is inpainted out of the support, which is regenerated and converted into a watertight collision mesh. A language model picks the part named in the prompt (seat, backrest, arm), and the object is dropped from the lowest intersection-free height above it.
One contact model. All bodies share an incremental-potential-contact barrier, so a state that starts intersection-free stays so.
Completion check and repair. Two geometric checks and a vision-language model judge the result. Failures are re-placed by grid search or by the adjoint of a differentiable simulation.

Results

Zero crossings and zero penetration on all 65 prompts. The rigid pipeline leaves cloth 14.8 mm inside the furniture, and Blender’s cloth solver crosses at 196 edges per thousand on the generated asset.
21 of 34 scenes are complete, against 8, 7, 2 and 0 for the baselines.
Held-out prompts: 19 of 32 complete with placement optimization, 14 from the physics stage alone.
Simulated manipulation: in all 23 pull, lift and drag episodes the gripped patch reaches 90% of its commanded displacement, median grip error 5.7 mm.

Limitations

Prompts that ask for an arrangement, such as a row of pillows or a folded towel, are treated as placements and fail the prompt check. A support generated as a solid block has no part to place onto. The sewn panel keeps the generated object’s kind in 48 of 60 scenes and its colour in 34. Materials are effective values, not measured ones, and plasticity is not modelled. Manipulation is simulated with a position constraint; the robot arm in the figures is for visualization only.

Figures

From Figure 1: SoftPAT3D generates simulation-ready scenes with deformable objects from text. The SoftPAT3D result, i... — From Figure 1: SoftPAT3D generates simulation-ready scenes with deformable objects from text. The SoftPAT3D result, in which every garment rests on the sofa and none intersects it.

From Figure 2: Overview of SoftPAT3D. (a) The prompt is expanded into a reference image, segmented and lifted as in P... — From Figure 2: Overview of SoftPAT3D. (a) The prompt is expanded into a reference image, segmented and lifted as in PAT3D, and the support hidden under each deformable object is inpainted, regenerated and converted into a collision mesh. (b) Each deformable object receives a rest shape from its manufacturing process under a size prior, a material from the material library and a placement above the part the prompt names. (c) One simulation brings the scene to equilibrium under a single contact barrier, and the result passes the geometric checks and the prompt check; a scene that fails enters placement optimization.

From Figure S1: Equilibrated scenes, rendered from the solver’s own output. Top: a blanket over a rocking chair, a kn... — From Figure S1: Equilibrated scenes, rendered from the solver’s own output. Top: a blanket over a rocking chair, a knit blanket over an armchair, a tablecloth on a square dining table, a gingham cloth on a picnic table. Bottom: a quilt on a daybed, a patterned throw over the back of a leather sofa, a sheet folded over the foot of a bed, and an apron hung over the back of a kitchen chair. Eight of the thirty-two scenes are shown.

From Figure 5: Vertical sections, cloth in red, furniture in grey, cloth inside the furniture in dark red, and a dash... — From Figure 5: Vertical sections, cloth in red, furniture in grey, cloth inside the furniture in dark red, and a dashed line where the cloth floats above its support. Top: GPT-6 Astra; the blanket and the towel hang above their seats, the quilt runs through the mattress, the pillow sits inside the backrest, and the skirt ripples in mid-air. Bottom: ours; the cloth rests on the support, wraps its edges and hangs from them.

Abstract

We present SoftPAT3D, a physics-augmented framework for text-to-3D scene generation in which deformable objects, such as cloth, stuffed toys and ropes, are generated as physically valid, simulation-ready bodies and placed in intersection-free contact with the furniture that supports them, with one another and with themselves. Existing physics-augmented pipelines simulate rigid bodies only and cannot accommodate a generated deformable object, which text-to-3D generation returns as a geometrically invalid mesh, sometimes self-intersecting, that has no rest shape, no material model and no admissible initial state. These are the inputs a deformable simulation requires, and we construct each from information already in the pipeline. The rest shape comes from how the object is manufactured rather than from the generated physically inconsistent deformed configuration, and a material library assigns each deformable object its initial elastic parameters, which can then be optimized. The occluded support is regenerated and converted into a watertight collision mesh, each object is released from an optimizable placement above the supporting part that the prompt specifies, and the complete scene is driven to static equilibrium under one unified contact model, with completion verified by geometric checks and a vision-language model. When a scene fails the prompt, a grid search over placements corrects it, and the adjoint of a differentiable simulation of the same contact energy reaches the targets the grid does not, places several bodies jointly and recovers and optimizes material parameters. Multi-body scenes equilibrate in a single simulation, and the pipeline also runs from the text prompt alone. On all sixty-five prompts, SoftPAT3D produces no undesired interpenetration of any kind, whereas every other system does, and its scenes directly support scene editing and robotic manipulation.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong