DebSDF: Delving into the Details and Bias of Neural Indoor Scene Reconstruction

TPAMI 2024

Yuting Xiao^1* · Jingwei Xu^1* · Zehao Yu² · Shenghua Gao^1†


ShanghaiTech University | University of Tübingen
(^* denotes equal contributions, † denotes corresponding author)

News

• [2024.7.26] Training data updated.
• [2024.6.12] Checkpoint release.
• [2024.6.11] Code release.
• [2024.6.6] Training data released.
• [2024.6.5] Accepted to TPAMI 2024. Congratulations to every author.

Demo

preview.mp4

More results are presented in the project page.

Setup

conda create -n DebSDF python=3.10
conda activate DebSDF

pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121

Dataset

For the preprocessed training data, please download from google drive.

#7 (comment): Please download this training data again, if you download the data before [2024.7.26]

Please extract the data to the ./data folder.

Our training data is adapted from MonoSDF. The ICL data please view from original author's webpage.

Some scene files have been provided by various artists for free on BlendSwap. Please refer to the table below for license information and links to the .blend files. (denoted as neural_rgbd_data in google drive)

License	Scene name
CC-BY	Breakfast room
CC-0	Complete kitchen
CC-BY	Green room
CC-BY	Grey-white room
CC-BY	Kitchen
CC-0	Morning apartment
CC-BY	Whiteroom

Run With Your Own Data

Please consider following the guidelines provided by MonoSDF.

Code

Scannet (scan_id from 1 to 4) running example:

cd code
python -m torch.distributed.launch  --master_port 22112 --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf ./confs/scannet_mlp.conf  --scan_id $scan_id

or you can run:

cd code
sh train.sh ./confs/scannet_mlp.conf 1 0
(means run the scan 1 of scannet with GPU 0)

BlenderSwap (scan_id from 1 to 7) running example:

cd code
python -m torch.distributed.launch  --master_port 22112 --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf ./confs/neural_rgbd_mlp.conf  --scan_id $scan_id

or you can run:

cd code
sh train.sh ./confs/neural_rgbd_mlp.conf 1 0
(means run the scan 1 of selected blender scenes with GPU 0)

Replica (scan_id from 1 to 8) running example:

cd code
python -m torch.distributed.launch  --master_port 22112 --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf ./confs/replica_mlp.conf  --scan_id $scan_id

or you can run:

cd code
sh train.sh ./confs/replica_mlp.conf 1 0
(means run the scan 1 of replica with GPU 0)

TNT Advanced(scan_id from 1 to 4) running example:

Please make sure the config file corresponds to the scan id.

cd code
python -m torch.distributed.launch  --master_port 22112 --nproc_per_node 1 --nnodes=1 --node_rank=0 training/exp_runner.py --conf ./confs/tnt_mlp_1.conf  --scan_id 1

or you can run:

cd code
sh train.sh ./confs/tnt_mlp_1.conf 1 0
(means run the scan 1 of tnt advanced with GPU 0)

sh train.sh ./confs/tnt_mlp_2.conf 2 1
(means run the scan 2 of tnt advanced with GPU 1)

Evaluation

Additional Setup

pip install -r evaluation_requirements.txt

Evaluate Scannet

cd scannet_eval
python3 evaluate.py

Evaluate BlenderSwap by artists (denoted as neural rgbd for convience)

cd neural_rgbd_eval
python3 evaluate.py

Evaluate Replica

cd replica_eval
python3 evaluate.py

Evaluate TNT Advanced

You need to submit the reconstruction results to the official evaluation server, please follow their guidance. We provide an example of our submission at this link.

Official Checkpoint Evaluation

Please download the corresponding checkpoint from this link.

And then extract the zip files to ./exps folder.

For example, please download scannet_exps.zip from this link. And the file tree of the extracted zip file is like:

├── scannet_mlp_1
│   └── _pretrained
│       ├── checkpoints
│       │   ├── ModelParameters
│       │   │   └── latest.pth
│       │   ├── OptimizerParameters
│       │   │   └── latest.pth
│       │   └── SchedulerParameters
│       │       └── latest.pth
│       └── runconf.conf
├── scannet_mlp_2
│   └── _pretrained
│       ├── checkpoints
│       │   ├── ModelParameters
│       │   │   └── latest.pth
│       │   ├── OptimizerParameters
│       │   │   └── latest.pth
│       │   └── SchedulerParameters
│       │       └── latest.pth
│       └── runconf.conf
├── scannet_mlp_3
│   └── _pretrained
│       ├── checkpoints
│       │   ├── ModelParameters
│       │   │   └── latest.pth
│       │   ├── OptimizerParameters
│       │   │   └── latest.pth
│       │   └── SchedulerParameters
│       │       └── latest.pth
│       └── runconf.conf
└── scannet_mlp_4
    └── _pretrained
        ├── checkpoints
        │   ├── ModelParameters
        │   │   └── latest.pth
        │   ├── OptimizerParameters
        │   │   └── latest.pth
        │   └── SchedulerParameters
        │       └── latest.pth
        └── runconf.conf

Please place the folder like:

.
├── README.md
├── code
├── evaluation_requirements.txt
├── exps
│   ├── scannet_mlp_1
│   ├── scannet_mlp_2
│   ├── scannet_mlp_3
│   └── scannet_mlp_4
├── media
├── neural_rgbd_eval
├── replica_eval
├── requirements.txt
└── scannet_eval

Then directly run:

sh scripts/scannet_checkpoint_inference.sh

Similarly, for BlenderSwap Scenes(Should be align with the metrics ICL-NUIM in paper table):

sh scripts/neural_rgbd_checkpoint_inference.sh

Similarly, for Replica:

sh scripts/replica_checkpoint_inference.sh

For TNT Advanced, please view here for our submitted files.

Acknowledgement

Our code is based upon MonoSDF. Additional training data is also processed with Omnidata for monocular depth and normal extraction.

Bibtex

@article{debsdf2024,
    title={DebSDF: Delving into the Details and Bias of Neural Indoor Scene Reconstruction},
    author={Yuting Xiao and Jingwei Xu and Zehao Yu and Shenghua Gao},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
    year={2024}
}