HTTP/2 301 server: GitHub.com content-type: text/html location: https://minfenli.github.io/GenRC/ access-control-allow-origin: * strict-transport-security: max-age=31556952 expires: Tue, 30 Dec 2025 06:47:44 GMT cache-control: max-age=600 x-proxy-cache: MISS x-github-request-id: C44B:234FE9:9CD341:B000A1:69537336 accept-ranges: bytes age: 0 date: Tue, 30 Dec 2025 06:37:44 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210045-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1767076664.440985,VS0,VE219 vary: Accept-Encoding x-fastly-request-id: 98445498a385149d703aebbde6ff57c9124e4712 content-length: 162 HTTP/2 200 server: GitHub.com content-type: text/html; charset=utf-8 last-modified: Fri, 13 Jun 2025 07:10:10 GMT access-control-allow-origin: * strict-transport-security: max-age=31556952 etag: W/"684bced2-4b78" expires: Tue, 30 Dec 2025 06:47:44 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: F37B:444BC:9C6496:AF907A:69537338 accept-ranges: bytes age: 0 date: Tue, 30 Dec 2025 06:37:44 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210045-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1767076665.688859,VS0,VE222 vary: Accept-Encoding x-fastly-request-id: 7ef1c8f828b65358fe61c2d86baf76835fb814bf content-length: 4402

GenRC: 3D Indoor Scene Generation
from Sparse Image Collections

ECCV 2024

Ming-Feng Li¹, Yueh-Feng Ku², Hong-Xuan Yen², Chi Liu⁴,
Yu-Lun Liu³, Albert Y. C. Chen⁴, Cheng-Hao Kuo⁴, Min Sun^2,4

¹Carnegie Mellon University, ²National Tsing Hua University,
³National Yang Ming Chiao Tung University, ⁴Amazon

arXiv Code

Given a sparse collection of RGBD images that capture a scene,
our method can generate complete room-scale 3D meshes with high-fidelity texture.

Abstract

Sparse RGBD scene completion is a challenging task especially when considering consistent textures and geometries throughout the entire scene. Different from existing solutions that rely on human-designed text prompts or predefined camera trajectories, we propose GenRC, an automated training-free pipeline to complete a room-scale 3D mesh with high-fidelity textures. To achieve this, we first project the sparse RGBD images to a highly incomplete 3D mesh. Instead of iteratively generating novel views to fill in the void, we utilized our proposed E-Diffusion to generate a view-consistent panoramic RGBD image which ensures global geometry and appearance consistency. Furthermore, we maintain the input-output scene stylistic consistency through textual inversion to replace human-designed text prompts. To bridge the domain gap among datasets, E-Diffusion leverages models trained on large-scale datasets to generate diverse appearances. GenRC outperforms state-of-the-art methods under most appearance and geometric metrics on ScanNet and ARKitScenes datasets, even though GenRC is not trained on these datasets nor using predefined camera trajectories.

Pipeline of GenRC

Pipeline of GenRC: (a) Firstly, we extract text embeddings as a token to represent the style of provided RGBD images via textual inversion. Next, we project these images to a 3D mesh. (b) Following that, we render a panorama from a plausible room center and use equirectangular projection to render various viewpoints of the scene from the panoramic image. Then, we propose E-Diffusion that satisfies equirectangular geometry to concurrently denoise these images and determine their depth via monocular depth estimation, resulting in a cross-view consistent panoramic RGBD image. (c) Lastly, we sample novel views from the mesh to fill in holes, resulting in a complete mesh.

Concepts of E-Diffusion

Multi-view diffusion with equirectangular geometry: (a) Given an incomplete panoramic image, we first obtain several incomplete perspective images via equirectangular projection. (b) To denoise a perspective image at i-th view for one step, we first denoise all images to clean images and warp all the images to i-th view to get an averaged image. Then, we add random noise back to the averaged image to get a perspective image which is denoised for one step. Note that while we use images in RGB space here for illustration, the entire process is operated in latent space.

Visual Comparison with RGBD2

Results on ScanNet

Comparison with Baselines on Scannet: GenRC can produce a comprehensive room-scale mesh with high-fidelity texture, even when provided with sparse RGBD observations. In comparison to the prior method RGBD2[1], GenRC excels in generating more complete meshes and high-fidelity images. Besides, while T2R+RGBD (adapted from Text2Room[2]) achieves high-fidelity texture, it may generate cross-view inconsistent geometry and artifacts.

BibTeX

      @article{ming2024GenRC,
        author  = {Ming-Feng Li, Yueh-Feng Ku, Hong-Xuan Yen, Chi Liu, Yu-Lun Liu, Albert Y. C. Chen, Cheng-Hao Kuo, Min Sun},
        title   = {GenRC: 3D Indoor Scene Generation from Sparse Image Collections},
        journal = {ECCV},
        year    = {2024}
      }
    

References

[1] Lei, Jiabao, Jiapeng Tang, and Kui Jia. "RGBD2: Generative Scene Synthesis via Incremental View Inpainting Using RGBD Diffusion Models." CVPR 2023.
[2] Höllein, Lukas, et al. "Text2room: Extracting textured 3d meshes from 2d text-to-image models." ICCV 2023.