Renjie Li1,4, Panwang Pan1†, Bangbang Yang1, Dejia Xu2, Shijie Zhou3, Xuanyang Zhang1, Zeming Li1, Achuta Kadambi3, Zhangyang Wang2, Zhengzhong Tu4, Zhiwen Fan2
1ByteDance
2UT Austin
3UCLA
4TAMU
Renjie Li1,4, Panwang Pan1†, Bangbang Yang1, Dejia Xu2, Shijie Zhou3, Xuanyang Zhang1, Zeming Li1, Achuta Kadambi3, Zhangyang Wang2, Zhengzhong Tu4, Zhiwen Fan2
Feel free to contact me (paulpanwang@gmail.com) or open an issue if you have any questions or suggestions.
- Complete documents
- Code release for Animation Phase
- Code release for Lifting Phase
- Testing 16 scene data set release
git clone git@github.com:ShadowIterator/4K4DGen.git
cd animating
conda env create -f environment.yml
cd ../4dlifting
conda env create -f environment.yml
conda activate 4dlifting
pip install submodules/diff-gaussian-rasterization-depth
pip install submodules/simple-knn
Please first set your working directory to ./animating
The testing panorama is in the Google Drive.
For Animating, please extract the data under the ./animating/data folder. To run on your own data, please organize the data as following:
data
|-- <your_own>
| |-- <scene1>.jpg
| |-- <scene2>.jpg
| |-- ...
|-- <your_own>_mask
| |-- <scene1>.png
| |-- <scene2>.png
| |-- ...
|-- <your_own>_config
| |-- <scene1>.json
| |-- <scene2>.json
| |-- ...
You can also refer to the example provided in the data folder.
Download the checkpoint in googledrive. Extract it at animating/pretrained.
Please first set your working directory to ./4dlifting
Put the data under the ./data folder. Organize the files as following:
data
|-- <scene1>
| |-- <scene1>.gif
| |-- <scene1>_mask.png
|-- <scene2>
|-- ...
You can also refer to the I2 folder provided as an example under the data folder.
Please download the checkpoints from googledrive. Place them under the ./pre_checkpoints folder
To run the animating phase, please run
cd ./animating
conda activate animating
# do animating
python gen4k.py --config ./svd_mask_config.yaml --eval validation_data.prompt_image=<path_to_image> validation_data.mask=<path_to_mask> validation_data.config=<path_to_config> pretrained_model_path=pretrained/animate_anything_svd_v1.0 --diff_mode return_latents
# decode latent codes
python gen4k.py --config ./svd_mask_config.yaml --eval validation_data.prompt_image=<path_to_image> validation_data.mask=<path_to_mask> validation_data.config=<path_to_config> pretrained_model_path=pretrained/animate_anything_svd_v1.0 --diff_mode decode_latents
You can also run the example bash file:
bash ./generate_example.sh
to run 4D lifting, please refer to the following.
cd 4dlifting
conda activate 4dlifting
# generate initial geometry for each frame
python generate_init_geo_4k.py -s <source_path> -m <target_path>
# lifting the frames
python train.py -s <source_path>/<frame_id> -m <target_path>/<frame_id>
using the script as following:
python run.py --input_img <your_image_path_or_folder>
If you have setup the data and environments as above, it will generate a running script run.sh. You can simply run bash ./run.sh to generate the 4D scene from your input images. For more options of the script, please see python run.py --help.
We built this from AnimateAnything, MultiDiffusion, and DreamScene360.
If you find our work useful for your project, please consider citing the following paper.
@misc{li20244k4dgenpanoramic4dgeneration,
title={4K4DGen: Panoramic 4D Generation at 4K Resolution},
author={Renjie Li and Panwang Pan and Bangbang Yang and Dejia Xu and Shijie Zhou and Xuanyang Zhang and Zeming Li and Achuta Kadambi and Zhangyang Wang and Zhengzhong Tu and Zhiwen Fan},
year={2024},
eprint={2406.13527},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2406.13527},
}
