| CARVIEW |
Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
ICCV 2023
*Work done during a remote internship with UCSD
Abstract
3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. Despite numerous task-specific methods, developing a comprehensive model remains challenging. In this paper, we present SSDNeRF, a unified approach that employs an expressive diffusion model to learn a generalizable prior of neural radiance fields (NeRF) from multi-view images of diverse objects. Previous studies have used two-stage approaches that rely on pretrained NeRFs as real data to train diffusion models. In contrast, we propose a new single-stage training paradigm with an end-to-end objective that jointly optimizes a NeRF auto-decoder and a latent diffusion model, enabling simultaneous 3D reconstruction and prior learning, even from sparsely available views. At test time, we can directly sample the diffusion prior for unconditional generation, or combine it with arbitrary observations of unseen objects for NeRF reconstruction. SSDNeRF demonstrates robust results comparable to or better than leading task-specific methods in unconditional generation and single/sparse-view 3D reconstruction.
During training, SSDNeRF jointly learns triplane features of individual scenes, a shared NeRF decoder, and a triplane diffusion prior. During testing, it can perform (a) unconditional generation, (b) single-view reconstruction, as well as multi-view reconstruction.
An overview of SSDNeRF framework with a triplane NeRF representation. During training, we feed a batch of observations in the format of pixel RGBs and rays. The corresponding scene code is randomly initialized and optimized by minimizing both the rendering loss and the diffusion loss, and model parameters ϕ, ψ are also updated along the way.
Unconditional Generation
Trained on ABO Tables
Trained on ShapeNet-SRN Cars
Single-View Reconstruction
Inputs from ShapeNet-SRN Cars test set
Inputs from ShapeNet-SRN Chairs test set
[Sim-to-real] Inputs from KITTI real images (trained on SRN Cars)
Sparse-to-Dense Reconstruction
Novel view synthesis quality (LPIPS) vs. number of input views, evaluated on SRN Cars.
BibTeX
@inproceedings{ssdnerf,
title={Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction},
author={Hansheng Chen and Jiatao Gu and Anpei Chen and Wei Tian and Zhuowen Tu and Lingjie Liu and Hao Su},
year={2023},
booktitle={ICCV}
}