PhysLDM: Latent Diffusion for
High-Fidelity Deformable Simulation

Yu Zhang1, Xudong Xu2, Xingang Pan1
1S-Lab, Nanyang Technological University, 2Shanghai AI Laboratory
Paper arXiv Video All Results

Abstract

Neural simulation of high-fidelity deformable bodies is a foundational challenge in computer graphics and physical AI. Long-horizon prediction for high-resolution 3D volumetric meshes is difficult: autoregressive methods are susceptible to error accumulation, while direct multi-frame prediction at native resolution is computationally prohibitive. This motivates a compact spatiotemporal latent representation, which remains largely unexplored for mesh-based volumetric physics. Meanwhile, it remains unclear whether deterministic regression or generative diffusion is the more appropriate predictive paradigm.

To address these coupled challenges, we introduce PhysLDM, a unified latent-diffusion paradigm for one-shot volumetric deformable simulation. Its core is a holistic spatiotemporal VAE (ST-VAE) that avoids the “staircase” artifacts of standard temporal compression (as in common video VAEs) and disentangles rigid motion from local deformation, achieving ∼2.48 mm reconstruction precision on meter-scale scenes at up to 78× token compression. Based on this reliable latent space, we systematically compare regression and diffusion paradigms. Our experiments uncover a key modeling insight: complex deformable dynamics are often chaotic, and in this regime deterministic regression tends to produce damped, non-physical averages, whereas diffusion better models their distribution. Accordingly, we employ a latent diffusion model that effectively learns from the chaotic data to generate physically plausible trajectories.

Trained purely kinematically on an Objaverse-scale simulation dataset, a single PhysLDM generalizes zero-shot to unseen OOD object datasets (GSO and Toys4K). Its differentiability further enables efficient solution of inverse problems and higher-order design optimization. To our knowledge, PhysLDM is the first high-fidelity spatiotemporal autoencoder and latent-diffusion paradigm for volumetric deformable dynamics, offering a scalable and robust approach to neural simulation.

~2.48 mm
reconstruction precision
on meter-scale scenes
78×
token compression
(up to)
64
frames predicted
in one diffusion pass

Reconstruction

A holistic spatiotemporal VAE compresses a whole 64-frame trajectory into one compact latent while keeping millimeter-scale geometric detail.

Ground truth versus ST-VAE reconstruction of a high-resolution soft mug
Ground truth (rose) versus ST-VAE reconstruction (blue) for an unseen high-resolution soft mug (11,621 nodes, >2× the training cap). The trajectory is compressed by ∼182× with only a 5.3 mm mean error. Far right: a cross-section of the final frame visualizing the internal tetrahedral structure.

Standard temporal compression, as in common video VAEs, loses high-frequency information and produces “staircase” artifacts: unnatural, abrupt temporal deformations. Directly compressing world coordinates entangles features across scales and fails to preserve geometric detail. Our holistic ST-VAE, which decouples rigid motion from deformation, removes the staircase problem and keeps the geometric detail.

Ground truth (left) and ST-VAE reconstruction (right) on held-out Objaverse scenes, followed by out-of-distribution (OOD) objects from GSO and Toys4K. The ST-VAE reconstructs the OOD objects zero-shot. Achieves ~2.48 mm precision on meter-scale scenes. The Young’s moduli were chosen to show both soft bodies with large deformation and stiff, nearly rigid ones, so some values do not match the real material of the object.

Cut-away views reveal the reconstructed interior tetrahedral nodes.

Complex unseen shapes, with the motion frozen at the moment of maximum deformation.

Five variants on the same scene: ground truth, Holistic (ours), Standard temporal compression, Standard + velocity loss, and direct world-coordinate compression. Each case is shown with the per-frame deformation change and a frame-by-frame silhouette difference.

Sixteen unseen cases. Rose: ground truth; blue: ST-VAE reconstruction.

ST-VAE reconstruction of two- and three-body scenes, each body with a random Young’s modulus and initial velocity.

All reconstruction videos →

Prediction

From the initial state, the latent diffusion model predicts all 64 frames in a single diffusion pass, with random stiffness and initial velocity.

Initial state (left) and predicted settled state (right) of 26 objects
One-shot prediction of 64-frame trajectories for diverse out-of-distribution (OOD) objects (GSO/Toys4K) across varying materials and initial velocities (left: initial state; right: predicted settled state).

Five reruns of the same scene with identical initial input on a GPU IPC solver end in significantly different states. In this regime deterministic regression tends to produce damped, non-physical averages, whereas diffusion better models their distribution.

Ground truth, our latent diffusion model, a capacity-matched regression model, and an MGN-style autoregressive baseline (shown as “GNS” in the video), from the same initial state.

Out-of-distribution (OOD) objects from GSO and Toys4K, with random stiffness and initial velocity. The diffusion model is applied zero-shot.

Harder unseen cases from held-out Objaverse and GSO.

Twenty-eight unseen cases from held-out Objaverse, GSO and Toys4K.

One-shot prediction on two- and three-body scenes, each body with a random Young’s modulus and initial velocity.

All prediction videos →

Differentiable Applications

PhysLDM is differentiable end to end, so gradients flow from a trajectory objective back to physical parameters and initial conditions.

Target trajectory (left) and recovering Young’s modulus by gradient descent (right)
A classic material estimation task: recovering Young’s modulus E (blue sequence) by matching a target state (textured) via gradient descent. Furthermore, designing the throw that best reveals the material is a second-order inverse task. PhysLDM easily solves it with native support for higher-order derivatives. By contrast, all other differentiable simulators we tested either fail or are computationally prohibitive on this task.

Given a single trajectory of a body with unknown stiffness, PhysLDM recovers its Young’s modulus by gradient descent through the model, within 0.06 dex in about 40 seconds on all seven test bodies. DiffIPC solves the task only after hand-tuning its solver and then needs 26 minutes on 48 threads; Newton/Warp fails in the backward pass.

Which throw reveals the stiffness best? We maximize the sensitivity of the predicted trajectory to the stiffness over the release velocity. The objective is itself a derivative, so its gradient is second-order; PhysLDM provides it directly through automatic differentiation. Replayed in the ground-truth simulator, the optimized throw makes the rigid motion 13× more sensitive to the stiffness. DiffIPC offers only first-order derivatives, and Newton/Warp does not produce a usable gradient on this scene.

All differentiable application videos →

Method

PhysLDM architecture: Spatial VAE, ST-VAE and latent diffusion model

Overview of PhysLDM. The framework consists of three main components: (1) A Spatial VAE (top left) that encodes the per-frame canonical mesh into a compact set of resolution-independent latent tokens. (2) An ST-VAE (bottom) that compresses the temporal trajectory into a single joint latent Z, which serves as the training target of the LDM. (3) An LDM (top right) that predicts the entire latent trajectory from the initial state in a single diffusion pass. Dec ∗ denotes the frozen decoders, which decode the latent into the final full trajectory.

Constitutive-Model-Agnostic Supervision

PhysLDM is trained with kinematic supervision only: no energy model or material law enters the data format, the losses, or the architecture. The ST-VAE therefore generalizes zero-shot across energy models (4.6 mm reconstruction error under ARAP) and to real-world data (6.0 mm reconstruction error).

Ground truth (left) and zero-shot ST-VAE reconstruction (right) under an as-rigid-as-possible (ARAP) energy. The ST-VAE was trained only on Stable Neo-Hookean simulations, so this energy model never appears in training.

Seamless support for other data formats, such as surface meshes and point clouds. Our training data are tetrahedral meshes, but PhysLDM runs on these formats without any change to the model or the training code. Shown here are real captured point clouds from PhysTwin: tracked points (left) and zero-shot ST-VAE reconstruction (right). There is no mesh and no explicit energy model, and the objects are manipulated by a human hand instead of being thrown onto the ground.

All constitutive-model-agnostic videos →

BibTeX

@article{zhang2026physldm,
  title   = {PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation},
  author  = {Zhang, Yu and Xu, Xudong and Pan, Xingang},
  journal = {arXiv preprint arXiv:2610.07609},
  year    = {2026}
}