ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback

We provide a demo website for you to play with our ControlNet++ models and generate images interactively. For local running, please run the following command:

git clone https://github.com/liming-ai/ControlNet_Plus_Plus.git
git checkout inference
pip3 install -r requirements.txt

Please download the model weights and put them into each subset of checkpoints:

And then run:

python3 app.py

We model image-based controllable generation as an image translation task from input conditional controls $c_v$ to output generated images $x'_0$. If we translate images from one domain to the other (condition $c_v$ → generated image $x'_0$ ), and back again (generated image $x'_0$ → condition $c_v'$ ) we should arrive where we started ($c_v$ = $c_v'$). Hence, we can directly optimize the cycle consistency loss for better controllability.

(a) Existing methods achieve implicit controllability by introducing imagebased conditional control $c_v$ into the denoising process of diffusion models, with the guidance of latent-space denoising loss. (b) We utilize discriminative reward models $D$ to explicitly optimize the controllability of $G$ via pixel-level cycle consistency loss.

(a) Pipeline of default reward fine-tuning strategy. Reward fine-tuning requires sampling all the way to the full image. Such a method needs to keep all gradients for each timestep and the memory required is unbearable by current GPUs. (b) We add a small noise ($t ≤ t_{thre}$) to disturb the consistency between input images and conditions, then the single-step denoised image can be directly used for efficient reward fine-tuning.

This project is licensed under the MIT License - see the LICENSE file for details.

If our work assists your research, feel free to give us a star ⭐ or cite us using:

@inproceedings{controlnet_plus_plus,
    author    = {Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, Chen Chen},
    title     = {ControlNet $$++ $$: Improving Conditional Controls with Efficient Consistency Feedback},
    booktitle = {European Conference on Computer Vision (ECCV)},
    year      = {2024},
}