| CARVIEW |
ReplaceAnything3D
Text-Guided 3D Scene Editing with Compositional Neural Radiance Fields
NeurIPS 2024
University College London, Alan Turing Institute
Reality Labs Research, Meta
Reality Labs Research, Meta
Reality Labs Research, Meta
Reality Labs Research, Meta
Reality Labs Research, Meta
Reality Labs Research, Meta
Reality Labs Research, Meta
Abstract
We introduce ReplaceAnything3D model (RAM3D), a novel text-guided 3D scene editing method that enables the replacement of specific objects within a scene. Given multiview images of a scene, a text prompt describing the object to replace, and text prompt describing the new object, our Erase-and-Replace approach can effectively swap objects in the scene with newly generated content while maintaining 3D consistency across multiple viewpoints. We demonstrate the versatility of ReplaceAnything3D by applying it to various realistic 3D scenes, showcasing results of modified foreground objects that are well-integrated with the rest of the scene without affecting its overall integrity.
Method
We distill a pretrained inpainting Latent Diffusion Model over 3 stages
- Erase stage: remove the masked objects and fill in the background
- Replace stage: generate new objects and composite them to the inpainted background scene
- Finally: train a new NeRF using the edited scene images
Visualise Scene edits
Click to play video, then drag the slider to compare the original and edited scene
Additional Results
Statue scene
Fern Scene
Spin-NeRF red net scene
Mip-NeRF Garden scene
Citation
If you find our work useful, please consider citing
@misc{bartrum2024replaceanything3dtextguided,
title={ReplaceAnything3D:Text-Guided 3D Scene Editing
with Compositional Neural Radiance Fields},
author={Edward Bartrum and Thu Nguyen-Phuoc and
Chris Xie and Zhengqin Li and Numair Khan and
Armen Avetisyan and Douglas Lanman and Lei Xiao},
year={2024},
eprint={2401.17895},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
