| CARVIEW |
DisCO: Portrait Distortion Correction with Perspective-Aware 3D GANs
Portrait photos captured from a near-range distance often suffer from undesired perspective distortions. DisCO corrects these perspective distortions and synthesizes more pleasant views by virtually enlarging focal length and camera-to-subject distance.
More results of cropped faces
We show that our method can process diverse in-the-wild face images
Correct in-the-wild distorted images collected by us and by [Zhao+, ICCV'19]
Abstract
Close-up facial images captured at short distances often suffer from perspective distortion, resulting in exaggerated facial features and unnatural/unattractive appearances. We propose a simple yet effective method for correcting perspective distortions in a single close-up face. We first perform GAN inversion using a perspective-distorted input facial image by jointly optimizing the camera intrinsic/extrinsic parameters and face latent code. To address the ambiguity of joint optimization, we develop starting from a short distance, optimization scheduling, reparametrizations, and geometric regularization. Re-rendering the portrait at a proper focal length and camera distance effectively corrects perspective distortions and produces more natural-looking results. Our experiments show that our method compares favorably against previous approaches qualitatively and quantitatively. We showcase numerous examples validating the applicability of our method on in-the-wild portrait photos. We will release our code and the evaluation protocol to facilitate future work.
Method Overview
Visual Comparisons
Visual comparisons on our collected images
Visual comparisons on images collected by [Zhao+, ICCV'19]
Quantitative comparisons
We conduct comparisons on CMDP. Results of [Fried+, TOG'16] is borrowed from their demo. -[Fried+, TOG'16] denotes our re-implementation of [Fried+, TOG'16].
Ablation study
Ablation study of our proposed perspective-aware 3D GAN inversion
Ablation study of our pipeline
Reference
- [Fried+, TOG'16] Ohad Fried, Eli Shechtman, Dan B Goldman, and Adam Finkelstein, Perspective-aware Manipulation of Portrait Photos, ACM TOG (Proc. SIGGRAPH), 2016.
- [Zhao+, ICCV'19] Yajie Zhao, Zeng Huang, Tianye Li, Weikai Chen, Chloe LeGendre, Xinglei Ren, Jun Xing, Ari Shapiro, and Hao Li, Learning Perspective Undistortion of Portraits, ICCV, 2019.
- Burgos-Artizzu, Xavier, Ronchi, Matteo Ruggero, & Perona, Pietro (2022). Caltech Multi-Distance Portraits (CMDP) (1.0) [Data set]. CaltechDATA. https://doi.org/10.22002/D1.20110
- [Shih+, CVPR'20] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, Jia-Bin Huang, 3D Photography using Context-aware Layered Depth Inpainting, CVPR, 2020.
- [Ko+, WACV'23] Jaehoon Ko, Kyusun Cho, Daewon Choi, Kwangrok Ryoo, Seungryong Kim, 3D GAN Inversion with Pose Optimization, WACV, 2023.
- [Tzaban+, SIGGRAPH Asia'22] Rotem Tzaban, Ron Mokady, Rinon Gal, Amit Bermano, Daniel Cohen-Or, Stitch it in Time: GAN-Based Facial Editing of Real Videos, SIGGRAPH Asia, 2022.
BibTeX
@article{wang2023disco,
title={DisCO: Portrait Distortion Correction with Perspective-Aware 3D GANs},
author={Zhixiang Wang, Yu-Lun Liu, Jia-Bin Huang, Shin'ichi Satoh, Sizhuo Ma, Guru Krishnan, Jian Wang},
journal={arXiv preprint arXiv:},
year={2023}
}Acknowledgements
Special thanks to Yajie Zhao for providing their results and data; Ohad Fried for sharing their results on web. Our collected in-the-wild images are from internet under common creative. Sources are here.