Jingtang Liang*, Xiaodong Cun*, Chi-Man Pun, Jue Wang

Paper | Code Repo | Demos (this repo) | Google Colab

Image harmonization aims to modify the color of the composited region with respect to the specific background. Previous works model this task as a pixel-wise image-to-image translation using UNet family structures. However, the model size and computational cost limit the performability of their models on edge devices and higher-resolution images. To this end, we propose a novel spatial-separated curve rendering network(S²CRNet) for efficient and high-resolution image harmonization for the first time. In S²CRNet, we firstly extract the spatial-separated embeddings from the thumbnails of the masked foreground and background individually. Then, we design a curve rendering module(CRM), which learns and combines the spatial-specific knowledge using linear layers to generate the parameters of the pixel-wise curve mapping in the foreground region. Finally, we directly render the original high-resolution images using the learned color curve. Besides, we also make two extensions of the proposed framework via the Cascaded-CRM and Semantic-CRM for cascaded refinement and semantic guidance, respectively. Experiments show that the proposed method reduces more than 90% parameters compared with previous methods but still achieves the state-of-the-art performance on both synthesized iHarmony4 and real-world DIH test set. Moreover, our method can work smoothly on higher resolution images in real-time which is more than 10× faster than the existing methods.

Our method runs very fast and can harmonized the image/videos in CPU in near real-time! We have tested the performance on both Linux/Mac M1.

Our method only requires minimal dependences, please check the requirements.txt.

pip install -r requirements.txt

We provide a demo.ipynb which shows the learned curve of each stage and the corresponding harmonized images, and the ground truth (target) from the sample dataset.

We use RobustVideoMatting for video matting and the local video camera as the source video.

python video_harmonization_demo.py

will create a side-by-side view to show the comparsion of the original video, copy-paste directly video and the harmonized video from left to right.

@misc{liang2021spatialseparated,
      title={Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization}, 
      author={Jingtang Liang and Xiaodong Cun and Chi-Man Pun and Jue Wang},
      year={2021},
      eprint={2109.05750},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

• [TIP 2020] S²AM: Improving the Harmony of the Composite Image by Spatial-Separated Attention Module.
• [AAAI 2021] Split then Refine: Sequential Attention-guided ResUNets for Blind Single Image Visible Watermark Removal
• [AAAI 2023] CoordFill: Efficient High-Resolution Image Inpainting via Parameterized Coordinate Querying (TODO)