DäRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation (NeurIPS 2023)

This is official implementation of the paper "DäRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation".

Introduction

Unlike existing work (SCADE [CVPR'23]) that distills depths by pretrained MDE to NeRF at seen view only, our DäRF fully exploits the ability of MDE by jointly optimizing NeRF and MDE at a specific scene, and distilling the monocular depth prior to NeRF at both seen and unseen views. For more details, please visit our project page!

TODO

• Reveal the pretrained-weight on Scannet
• TNT/in-the-wild datasets and dataloaders

Installation

An example of installation is shown below:

git clone https://github.com/KU-CVLAB/DaRF.git
cd DaRF
conda create -n DaRF python=3.8
conda activate DaRF
pip install -r requirements.txt
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

Also, you need to download pretrained MiDaS 3.0 weights(dpt_hybrid_384) on here.

And you should replace the 'dpt_pretrained_weight' part of the config file with the MiDaS pretrained weights path.

Dataset Download

You can download Scannet Dataset on here.

If you want to download data on a different path, you should replace the 'data_dirs' part of the config file with the donloaded dataset path.

Training

• 18-20 view Training

PYTHONPATH='.' python plenoxels/main.py --config plenoxels/configs/07XX.py

• 9-10 view Training

PYTHONPATH='.' python plenoxels/main.py --config plenoxels/configs/07XX_few.py

Evaluation / Rendering

If you want to Evaluation or Rendering, You need to replace the 'checkpoint' part of the config file with the trained weights path.

• 18-20 view Evalutaion

PYTHONPATH='.' python plenoxels/main.py --config plenoxels/configs/07XX.py --validate-only --load_model

• 18-20 view Rendering

PYTHONPATH='.' python plenoxels/main.py --config plenoxels/configs/07XX.py --render-only --load_model

• 9-10 view Evalutaion

PYTHONPATH='.' python plenoxels/main.py --config plenoxels/configs/07XX_few.py --validate-only --load_model

• 9-10 view Rendering

PYTHONPATH='.' python plenoxels/main.py --config plenoxels/configs/07XX_few.py --render-only --load_model

Acknowledgements

This code heavily borrows from K-planes.

Citation

If you use this software package, please cite our paper:

@article{song2023d,
  title={D$\backslash$" aRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation},
  author={Song, Jiuhn and Park, Seonghoon and An, Honggyu and Cho, Seokju and Kwak, Min-Seop and Cho, Sungjin and Kim, Seungryong},
  journal={arXiv preprint arXiv:2305.19201},
  year={2023}
}