Fangneng Zhan, Jiahui Zhang, Yingchen Yu, Rongliang Wu, Shijian Lu
Nanyang Technological University, Singapore

Abstract: This paper presents MoNCE, a versatile metric that introduces image contrast to learn a calibrated metric for the perception of multifaceted inter-image distances. Unlike vanilla contrast which indiscriminately pushes negative samples from the anchor regardless of their similarity, we propose to re-weight the pushing force of negative samples adaptively according to their similarity to the anchor, which facilitates the contrastive learning from informative negative samples. Since multiple patch-level contrastive objectives are involved in image distance measurement, we introduce optimal transport in MoNCE to modulate the pushing force of negative samples collaboratively across multiple contrastive objectives.

• Linux or macOS
• Python3, PyTorch
• CPU or NVIDIA GPU + CUDA CuDNN

Please follow the guidance in SPADE and CUT.

The pretrained model on Cityscapes, Horse2Zebra, Winter2Summer can be downloaded from Google Drive. Put them into CUT_MoNCE/checkpoints and run the command

cd CUT_MoNCE
bash test_cityscapes.sh

The pretrained model on ADE20K, CelebA-HQ (semantic), DeepFashion can be downloaded from Google Drive. Put them into SPADE_MoNCE/checkpoints and run the command

cd SPADE_MoNCE
bash test_ade20k.sh

Run the command

cd CUT_MoNCE
bash train_cityscapes.sh

Run the command

cd SPADE_MoNCE
bash train_ade20k.sh

If you use this code for your research, please cite our papers.

@inproceedings{zhan2022modulated,
  title={Modulated contrast for versatile image synthesis},
  author={Zhan, Fangneng and Zhang, Jiahui and Yu, Yingchen and Wu, Rongliang and Lu, Shijian},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={18280--18290},
  year={2022}
}

This code borrows heavily from CUT and SPADE.