Predicted 64-frame trajectories for 26 objects from GSO, Toys4K and held-out Objaverse scenes, thrown onto a stage with Young’s moduli from 3×104 to 3×106 Pa. Each object is predicted independently from its initial state.
Standard temporal compression, as in common video VAEs, loses high-frequency information and produces “staircase” artifacts: unnatural, abrupt temporal deformations. Directly compressing world coordinates entangles features across scales and fails to preserve geometric detail. Our holistic ST-VAE, which decouples rigid motion from deformation, removes the staircase problem and keeps the geometric detail.
Ground truth (left) and ST-VAE reconstruction (right) on held-out Objaverse scenes, followed by out-of-distribution (OOD) objects from GSO and Toys4K. The ST-VAE reconstructs the OOD objects zero-shot. Achieves ~2.48 mm precision on meter-scale scenes. The Young’s moduli were chosen to show both soft bodies with large deformation and stiff, nearly rigid ones, so some values do not match the real material of the object.
Cut-away views reveal the reconstructed interior tetrahedral nodes.
Complex unseen shapes, with the motion frozen at the moment of maximum deformation.
Five variants on the same scene: ground truth, Holistic (ours), Standard temporal compression, Standard + velocity loss, and direct world-coordinate compression. Each case is shown with the per-frame deformation change and a frame-by-frame silhouette difference.
Sixteen unseen cases. Rose: ground truth; blue: ST-VAE reconstruction.
ST-VAE reconstruction of two- and three-body scenes, each body with a random Young’s modulus and initial velocity.
Five reruns of the same scene with identical initial input on a GPU IPC solver end in significantly different states. In this regime deterministic regression tends to produce damped, non-physical averages, whereas diffusion better models their distribution.
Ground truth, our latent diffusion model, a capacity-matched regression model, and an MGN-style autoregressive baseline (shown as “GNS” in the video), from the same initial state.
Out-of-distribution (OOD) objects from GSO and Toys4K, with random stiffness and initial velocity. The diffusion model is applied zero-shot.
Harder unseen cases from held-out Objaverse and GSO.
Twenty-eight unseen cases from held-out Objaverse, GSO and Toys4K.
One-shot prediction on two- and three-body scenes, each body with a random Young’s modulus and initial velocity.
Given a single trajectory of a body with unknown stiffness, PhysLDM recovers its Young’s modulus by gradient descent through the model, within 0.06 dex in about 40 seconds on all seven test bodies. DiffIPC solves the task only after hand-tuning its solver and then needs 26 minutes on 48 threads; Newton/Warp fails in the backward pass.
Which throw reveals the stiffness best? We maximize the sensitivity of the predicted trajectory to the stiffness over the release velocity. The objective is itself a derivative, so its gradient is second-order; PhysLDM provides it directly through automatic differentiation. Replayed in the ground-truth simulator, the optimized throw makes the rigid motion 13× more sensitive to the stiffness. DiffIPC offers only first-order derivatives, and Newton/Warp does not produce a usable gradient on this scene.
Ground truth (left) and zero-shot ST-VAE reconstruction (right) under an as-rigid-as-possible (ARAP) energy. The ST-VAE was trained only on Stable Neo-Hookean simulations, so this energy model never appears in training.
Seamless support for other data formats, such as surface meshes and point clouds. Our training data are tetrahedral meshes, but PhysLDM runs on these formats without any change to the model or the training code. Shown here are real captured point clouds from PhysTwin: tracked points (left) and zero-shot ST-VAE reconstruction (right). There is no mesh and no explicit energy model, and the objects are manipulated by a human hand instead of being thrown onto the ground.