Urban scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce **DynamicCity**, a novel 4D occupancy generation framework capable of generating large-scale, high-quality dynamic 4D scenes with semantics. DynamicCity mainly consists of two key models: 1. A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel **Projection Module** to effectively compress 4D features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to **12.56** mIoU gain). Furthermore, we utilize an **Expansion & Squeeze Strategy** to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to **7.05** mIoU gain, **2.06x** training speedup, and **70.84%** memory reduction). 2. A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a **Padded Rollout Operation** is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting **versatile 4D generation applications**, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D occupancy generation methods across multiple metrics. The code and models have been released to facilitate future research.

conda create -n dyncity python=3.10 -y
conda activate dyncity
conda install pytorch==2.4.0 pytorch-cuda=11.8 -c pytorch -c nvidia
conda install einops hydra-core matplotlib numpy omegaconf timm tqdm wandb -c conda-forge -y
pip install flash-attn --no-build-isolation

DynamicCity
├── carlasc/
│   ├── Cartesian/
│   │   ├── Train/
│   │   │   ├── Town01_Heavy
│   │   │   ├── ...
│   │   ├── Test/
├── ...

torchrun --nproc-per-node 8 train.py VAE carlasc name=DYNAMIC_CITY_VAE

torchrun --nproc-per-node 8 infer_vae.py -n DYNAMIC_CITY_VAE --save_rollout --best

torchrun --nproc-per-node 8 train.py DiT carlasc name=DYNAMIC_CITY_DIT vae_name=DYNAMIC_CITY_VAE

torchrun --nproc-per-node 8 infer_dit.py -d DYNAMIC_CITY_DIT --best_vae

@inproceedings{bian2025dynamiccity,
  title={DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes},
  author={Bian, Hengwei and Kong, Lingdong and Xie, Haozhe and Pan, Liang and Qiao, Yu and Liu, Ziwei},
  booktitle={Proceedings of the International Conference on Learning Representations (ICLR)},
  year={2025},
}