1. Clone this repository:

git clone https://github.com/codepassionor/Tokenflow_adapter.git

2. Install the dependencies:

conda create --name myenv python=3.10
conda activate myenv
pip install -r requirements.txt

Before starting the training process, ensure the dataset is correctly prepared. Please download the MSRVTT dataset locally first, you can get it through this link MRSVTT.

Then use the following script to organize and preprocess your data for training:

python make_dataset/msrvtt-depth-map.py

This script handles dataset cleaning, and formatting to meet the training requirements.

To train ControlNet using a single GPU, you can directly run the following script:

bash train_controlnet.sh

For multi-GPU training, make the necessary modifications in train_controlnet.sh, and refer to the example below:

export CUDA_VISIBLE_DEVICES=0,1,2,3
accelerate launch     \
                    --num_machines 1  \
                    --num_processes 1 \
                    --gpu_ids 1       \
                    --num_cpu_threads_per_process 1 \
    main_controlnet.py 
--pretrained_model_name_or_path /data/workspace/huggingface/hub/runwayml/stable-diffusion-v1-5        --rank 4  --train_batch_size 1  --mixed_precision no

Use the script below for DDPM training:

bash train_multigpu.sh

To accelerate web connections, the train_ddpm.sh script uses proxychains. If you encounter connection issues, verify your proxy settings. For multi-GPU training, you can modify the script and use a similar setup as shown in the ControlNet example above.

This section shows some visualization results on downstream algorithms.

We provide a comparison of the effects of different timesteps of the training and inference process for you to pick the desired parameters.

• Testing Code
• Open-Sourcing Much More Visualization Results
• Project webpage
• More detailed running instructions