|
|
| CARVIEW |
We study the problem of generating temporal object intrinsics --- temporally evolving sequences of object geometry, reflectance, and texture, such as a blooming --- from pre-trained 2D foundation models.
What are the applications?
Using the generated temporal object intrinsics, we can insert them to any 3D scene, render them from any viewpoint, under any environmental lighting conditions, at any time of their lifespan.
Let's watch roses' birth and death:
Starting from a bud, the roses gradually expands their pedals and blossoms into full glory, only to wither and conclude their lifespans.
How is bread baked from fresh dough?
Young plants are sprouting...
Icecreams are melting...
Candles are burning...
Generated Temporal Object Intrinsics
We show the generated temporal object intrinsics using the proposed pipeline in this section.
Select Natural Phenomena:
Rendering
Geometry
Albedo
Material
Comparison w/ Baselines
We show the comparison of the generated 4D contents and baselines. Note that none of the baselines could generate object intrinsics or relightable graphics assets. Therefore we only compare the rendering.
Select Natural Phenomena:
View 1
View 2
Approach
For a modeled natural process, we first estimate a task-specific Neural Template to get a 4D-consistent representation of its temporal state changes. Then we iteratively optimize a hybrid 4D intrinsics field to generate temporal object intrinsics. In each iteration of optimization, we sample a camera pose and timestamp to render this optimizable 4D field into intrinsic maps. These maps are subsequently rendered into RGB images using a physically-based renderer. Concurrently, the same camera pose and timestamp are used to query the Neural Template for the corresponding Neural State Map. This map conditions a fine-tuned 2D diffusion model, which provides guidance signals to update the 4D representation. For more details, please refer to our paper.
Bibtex
@InProceedings{Geng2025Rose,
author = {Geng, Chen and Zhang, Yunzhi and Wu, Shangzhe and Wu, Jiajun},
title = {Birth and Death of a Rose},
booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
month = {June},
year = {2025},
pages = {26102-26113}
}