4D-LRM:
Large Space-Time Reconstruction Model From and To Any View at Any Time

Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at some times to any view at any time? We provide an affirmative answer with 4D-LRM, the first large-scale 4D reconstruction model that takes input from unconstrained views and timestamps and renders arbitrary novel view-time combinations. Unlike prior 4D approaches, e.g., optimizationbased, geometry-based, or generative, that struggle with efficiency, generalization, or faithfulness, 4D-LRM learns a unified space-time representation and directly predicts per-pixel 4D Gaussian primitives from posed image tokens across time, enabling fast, high-quality rendering at, in principle, infinite frame rate. Our results demonstrate that scaling spatiotemporal pretraining enables accurate and efficient 4D reconstruction. We show that 4D-LRM generalizes to novel objects, interpolates across time, and handles diverse camera setups. It reconstructs 24-frame sequences in one forward pass with less than 1.5 seconds on a single A100 GPU.

1. 4D-LRM-Base: Transformer with 768 hidden dimensions, 12 layers, and 12 attention heads, trained with 12 random input views and 12 random target views. No free Gaussians.
2. #Target x 2: Trained with 12 random input views and 24 random target views.
3. w/ Hexplane: Instead of unified space-time representation, an alternative 4DGS representation with decomposed neural voxel encoding inspired by HexPlane.
4. w/ Temp Align: Similar to the idea of pixel-aligned Gaussians, we force μ_T to the input frame time, reducing the parameterization to dim_4DGS = 19.
5. w/ Free GS: Trained with N = 1024 free Gaussian tokens from scratch.