LINK : https://arxiv.org/abs/2112.00374

@article{kwon2021clipstyler,
  title={Clipstyler: Image style transfer with a single text condition},
  author={Kwon, Gihyun and Ye, Jong Chul},
  journal={arXiv preprint arXiv:2112.00374},
  year={2021}
}

Pytorch 1.7.1, Python 3.6

$ conda create -n CLIPstyler python=3.6
$ conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0
$ pip install ftfy regex tqdm
$ conda install -c anaconda git
$ pip install git+https://github.com/openai/CLIP.git

We provide demo with replicate.ai

To train the model and obtain the image, run

python train_CLIPstyler.py --content_path ./test_set/face.jpg \
--content_name face --exp_name exp1 \
--text "Sketch with black pencil"

To change the style of custom image, please change the --content_path argument

edit the text condition with --text argument

For easy demo, we provide Google Colab .

*Warning : Due to slow computation speed of colab, it may take several minutes in colab environment

Before training, plase download DIV2K dataset LINK.

We recommend to use Training data of High-Resolution(HR) images.

To train the model, please download the pre-trained vgg encoder & decoder models in LINK.

Please save the downloaded models in ./models directory

Then, run the command

python train_fast.py --content_dir $DIV2K_DIR$ \
--name exp1 \
--text "Sketch with black pencil" --test_dir ./test_set

Please set the $DIV2K_DIR$ as the directory in which DIV2K images are saved.

To test the fast style transfer model,

python test_fast.py --test_dir ./test_set --decoder ./model_fast/clip_decoder_iter_200.pth.tar

Change the argument --decoder to other trained models for testing on different text conditions.

We provide several fine-tuned decoders for several text conditions. LINK

To use high-resolution image, please add --hr_dir ./hr_set to test command.

We provide colab notebook for testing fast transfer model

Style interpolation results with interpolating weight parameters of two fine-tuned decoder models

To interpolate the fast style transfer model,

python test_intp.py --decoder_src $SOURCE_DECODER --decoder_trg $TARGET_DECODER

Put source and target decoder model paths in $SOURCE_DECODER and $TARGET_DECODER

Style interpolation example with interpolating two styles "Stone wall" and "Desert sand"

For video style transfer, first install video io package

$ pip install imageio-ffmpeg
$ conda install -c conda-forge/label/cf202003 opencv

Then run the following command,

python test_video.py --content_path $VIDEO_PATH$ --decoder $DECODER_PATH$