[ICCV 2023] Multi-task View Synthesis with Neural Radiance Fields

Paper

Website

This repository contains a PyTorch implementation of our paper "Multi-task View Synthesis with Neural Radiance Fields".

To install the dependencies, follow the official repository of GeoNeRF:

The datasets are hosted on HuggingFace. "replica_training_views.zip" and "scenenet_training_views.zip" are used for training in the main experiments. "replica_training_full.zip" and "scenenet_training_full.zip" are used for held-in testing on novel views of already seen scenes. "replica_testing_full.zip" and "scenenet_testing_full.zip" are used for novel scene evaluation within the same dataset (in-distribution). "llff_full.zip", "tartan_full.zip", "scannet_full.zip", "blended_full.zip" are used for out-of-distribution scene evaluation.

Our training process contains two stages following cross-stitch networks. In the first stage, we train all the parameters except for the self-attention modules in the CTA module for 5,000 iterations. Run the command:

bash train_first_stage_scripts.sh

Afterwards, before proceeding to the second stage, change the path of the loaded pretrained weight on L764 in run_geo_nerf.py. Then run the command:

bash train_second_stage_scripts.sh

Before testing, change the path of the loaded pretrained weight on L778. Running the following scripts will start testing the multi-task view synthesis task on three novel scenes.

bash replica_test_scripts.sh

If you find our work useful, please consider citing:

@inproceedings{zheng2023mtvs,
  title={Multi-task View Synthesis with Neural Radiance Fields},
  author={Zheng, Shuhong and Bao, Zhipeng and Hebert, Martial and Wang, Yu-Xiong},
  booktitle={IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2023}
}

The codes are largely borrowed from the PyTorch implementation of GeoNeRF:

https://github.com/idiap/GeoNeRF

This work was supported in part by NSF Grant 2106825, Toyota Research Institute, NIFA Award 2020-67021-32799, the Jump ARCHES endowment, the NCSA Fellows program, the Illinois-Insper Partnership, and the Amazon Research Award. This work used NVIDIA GPUs at NCSA Delta through allocations CIS220014 and CIS230012 from the ACCESS program.